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PREFACE 


This  volume  is  part  of  a  four-volume  set  that  summarizes  the  research  of  participants  in 
the  1998  AFOSR  Summer  Research  Extension  Program  (SREP).  The  current  volume, 
Volume  1  of  5,  presents  the  final  reports  of  SREP  participants  at  Armstrong  Laboratory. 

Reports  presented  in  this  volume  are  arranged  alphabetically  by  author  and  are  numbered 
consecutively  --  e  g.,  1-1,  1-2,  1-3;  2-1,  2-2,  2-3,  with  each  series  of  reports  preceded  by 
a  35  page  management  summary.  Reports  in  the  five- volume  set  are  organized  as  follows: 
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Operations  and  Reports,  1215  Jefferson  Davis  Highway,  Suite  1204,  Arlington,  VA  222024302,  and  to  the  Office  of  Management  and  Budget,  P 


'  AFRL-SR-BL-TR-00- 


1.  AGENCY  USE  ONLY  (Leave blank) 


4.  TITLE  AND  SUBTITLE 


I  2.  REPORT  DATE 


December,  1998 


1998  Summer  Research  Program  (SRP),  Summer  Research  Extension  Program 
(SREP),  Final  Report,  Volume  3,  Rome  Laboratory 


S.  FUNDING  NUMBERS 

F49620-93-C-0063 


6.  AUTHOR(S) 
Gary  Moore 


7.  PERFORMING  ORGANIZATION  NAME(S)  AND  ADDRESS(ES) 
Research  &  Development  Laboratories  (RDL) 
5800  Uplander  Way 
Culver  City,  CA  90230-6608 


8.  PERFORMING  ORGANIZATION 
REPORT  NUMBER 


9.  SPONSORINGfMONITORING  AGENCY  NAME(S)  AND  ADDRESS(ES) 

Air  Force  Office  of  Scientific  Research  (AFOSR) 
801  N.  Randolph  St. 

Arlington,  VA  22203-1977 


10.  SPONSORINGfMONITORING 
AGENCY  REPORT  NUMBER 


12a.  DISTRIBUTION  AVAILABILITY  STATEMENT 

Approved  for  Public  Release 


13.  ABSTRACT  l, Maximum  200  words) 

The  United  States  Air  Force  Summer  Research  Program  (SRP)  is  designed  to  introduce  university,  college,  and  technical 
institute  faculty  members  to  Air  Force  research.  This  is  accomplished  by  the  faculty  members,  graduate  students,  and  high 
school  students  being  selected  on  a  nationally  advertised  competitive  basis  during  the  summer  intersession  period  to  perforn 
research  at  Air  Force  Research  Laboratory  (AFRL)  Technical  Directorates  and  Air  Force  Air  Logistics  Centers  (ALC). 
AFOSR  also  offers  its  research  associates  (faculty  only)  an  opportunity,  under  the  Summer  Research  Extension  Program 
(SREP),  to  continue  their  AFOSR-sponsored  research  at  their  home  institutions  through  the  award  of  research  grants.  This 
volume  consists  of  a  listing  of  the  participants  for  the  SREP  and  the  technical  report  from  each  participant  working  at  the  AI 
Rome  Laboratory. 


14.  SUBJECT  TERMS 

Air  Force  Research,  Air  Force,  Engineering,  Laboratories,  Reports,  Summer,  Universities, 
Faculty,  Graduate  Student,  High  School  Student 


1 15.  NUMBER  OF  PAGES 


116.  PRICE  CODE 


17.  SECURITY  CLASSIFICATION 
OF  REPORT 

Unclassified 


18.  SECURITY  CLASSIFICATION 
OF  THIS  PAGE 

Unclassified 


19.  SECURITY  CLASSIFICATION 
OF  ABSTRACT 

Unclassified 


20.  LIMITATION  OF 
ABSTRACT 


Standard  Form  298  (Rev.  2*89  EG 

Prescribed  by  ANSI  Std.  239.18 

Designed  using  Perform  Pro,  WHS/DIOR,  Oct  94 


GENERAL  IRISTRUCTIOHIS  FOR  COMPLETIHIG  SF  298 


The  Report  Documentation  Page  (RDP)  is  used  in  announcing  and  cataloging  reports.  It  is  important  that  this  information  be  consistent  with 
the  rest  of  the  report,  particularly  the  cover  and  title  page.  Instructions  for  filling  in  each  block  of  the  form  follow.  It  is  important  to  stay 
within  the  lines  to  meet  optical  scanning  requirements. 


Standard  Form  298  Back  (Rev.  2-89) 


1998  SREP  Final  Technical  Report  Table  of  Contents 


Armstrong  Research  Laboratory 
Volume  1 


Principle  Report  Title 

Investigator _ University/Institution 


1  Dr.  Gerald  P.  Chubb 

2  Dr.  Brent  D.  Foy 

3  Dr.  Charles  Lance 


Scoring  Pilot  Performance  of  Basic  Flight  Maneuvers 
Ohio  University 

Development  &  Validation  of  a  Physiologically-Based  Kinetic 
Model  of  Perfused  Liver  for  water-soluble  Compounds 
Wright  State  University 

Extension  of  Job  Performance  Measurement  Technologies  to 
Development  of  a  Prototype  Methodology  for  Assessing  Work  Team 
University  of  Georgia  Research  Foundation 


Dr.  David  Woelir  Validation  of  the  Multidimensional  Work  Ethic  Profile  (MWEP) 

as  a  Screening  Tool  for  AF  Enlisted  Personnel 
Texas  A  &  M  University  College  Station 


ii 


Laboratory  & 
Directorate 

AFRL/HEA 

AFRL/HES 

AFRL/HEJ 

AFRL/HEJ 


Principle 

Investigator 

Dr.  Mark  J.  Balas 

Dr.  Neb  Duric 

Dr.  George  W.  Hanson 

Dr.  Brian  D.  Jeffs 

Dr.  Aravinda  Kar 

Dr.  Donald  J.  Leo 

Dr.  Hanli  Liu 

Dr.  Joshua  C.  Biefang 
Dr.  Eric  J.  Paulson 

Dr.  Kenneth  F.  Stephens  II 


1998  SREP  Final  Technical  Report  Table  of  Contents 

Phillips  Research  Laboratory 
Volume  2 


Report  Title 
University/Institution 


Non-Linear  Adaptive  Control  for  a  Precision 
Deployable  Structure  with  White  ligh 
University  of  Colorado  at  Boulder 


Image  Recovery  Using  Phase  Diversity 
University  of  New  Mexico 


Perturbation  Analysis  of  the  Natural 
Frequencies  Targets  in  Inhomogeneous 
University  of  Wisconsin-Milwaukee 

Bayesian  restoration  of  Space  object  Images  from 
Adaptive  Optics  Data  with  unknown  data 
Brigham  Young  University 


Effects  of  Vapor-Plasma  Layer  on  Thick-Section 
Cutting  and  Calculation  of  Modes 
University  of  Central  Florida 

Adaptive  Vibration  suppression  for  autonomous 
Control  Systems 
University  of  Toledo 

Continuous-  Wave  approach  to  3-D  imaging 
Through  Turbid  media  w/a  Single  Planar 
Measurement 

University  of  Texas  Arlington 

Optical  Clocks  Based  on  Diode  Lasers 
University  of  New  Mexico 

Optimization  7  Analysis  of  a  Waverider  Vehicle 
For  Global  Spaceplane 
University  of  Colorado  at  Boulder 

Simulation  of  an  explosively  Formed  Fuse  Using 
MACH  2 

University  of  North  Texas 


Laboratory  & 
Directorate 

AFRL/VSDD 

AFRL/DEBS 

AFRL/DEHP 

AFRL/DES 

AFRL/DEOB 

AFRL/VSDV 

AFRL/DEBS 

AFRL/DELO 

AFRL/PRR 

AFRJL/DEHE 


1998  SREP  Final  Technical  Report  Table  of  Contents 


Rome  Research  Laboratory 
Volume  3 


Principle 

Investigator 


Report  Title 
University/Institution 


1  Dr.  Milica  Baijaktarovic 

2  Dr.  Stella  N.  Batalama 

3  Dr.  Nikolaos  G.  Bouibakis 

4  Dr.  Venugopala  R.  Dasigi 

5  Dr.  Richard  R.  Eckert 

6  Dr.  Kuo-Chi  Lin 

7  Dr.  Dimitrios  N.  Pados 

8  Dr.  Brajendra  N.  Panda 

9  Dr.  Michael  A  Pittarelli 

10  Dr.  Mark  S.  Schmalz 

11  Dr.  Nong  Ye 

12  Mr.  Parker  Bradley 


Specification  and  Verification  of  SDN.  701  MSP 
Functions  and  Missi  Crypto 
Wilkes  University 

Robust  Spread  Spectrum  Communications:  Adaptive 
Interference  Mitigation 
SUNY  Buffalo  University 

Hierarchical-Adaptive  Image  Segmentation 
SUNY  Binghamton  University 

Information  Fusion  w/Multiple  Feature  Extractors 
for  automatic  Text 
Sacred  Heart  University 

The  Interactive  Learning  Wall;  A  PC-Based,  Deployable  Data 
Wall  for  Use  in  a  College  Classroom 
SUNY  Binghamton  University 

Web-Based  Distributed  Simulation 
University  of  Central  Florida 

Adaptive  Array  Radars  and  Joint  Space-Time  Auxiliary 
Verctor  Filtering 

Information  Warfare”  Design  of  an  Efficient  Log  Management 
Method  to  Aid  In  Data 
University  of  North  Dakota 

Complexity  of  Detecting  and  content-driven  methods  for  resolving 
database 

SUNY  of  Tech  Utica 

Errors  Inherent  in  3D  Target  Reconstruction  from  Multiple 
Airborne  Images 
University  of  Florida 

Model-based  Assessment  of  Campaign  Plan-Performance 
under  Uncertainty 
Arizona  State  University 

Development  of  User-Friendly  CompEnvironment  for  Blind 
Source  Separation 
Syracuse  University 


iv 


Laboratory  & 
Directorate 

AFRL/IFGB 

AFRL/IFGC 

AFRL/IRE 

AFRL/IRE 

AFRL/EFSA 

AFRL/IFSB 

AFRL/SN 

AFRL/IFGB 

AFRL/tFTB 

AFRL/IRE 

AFRL/EFSA 

AFRL/IFGC 


1998  SREP  Final  Technical  Report  Table  of  Contents 


Wright  Research  Laboratory 
Volume  4 


Principle 

Investigator 


Report  Title 
University/Institntion 


1  Dr.  Brian  P.  Beecken 


2  Dr.  John  H.  Beggs 


3  Dr.  Raj  Bhatnagar 


Development  of  a  statistical  Model  predicting  the  impact  of 
a  scent  Projector's  Nonunifomiity  on  a  test  Article’s  Image 
Bethel  College 

Implementation  of  an  Optimization  Algorithm  in  Electromagnetics 
for  Radar  absorbing  Material  Layers 
Mississippi  State  University 

Analysis  of  Intra-Class  Variability  and  synthetic  Target  Models 
Models  for  Use  in  ATR 
University  of  Cincinnati 


4  Dr.  Gregory  Blaisdell 


5  Dr.  John  Douglas 

6  Dr.  William  Hosford 


7  Dr.  Yi  Pan 


8  Dr.  Kishore  Pochiraju 


9  Dr.  Yuri  Shtessel 


10  Dr.  Janusz  Starzyk 


Validation  of  a  Large  Eddy  Simulation  Code  &  Development  of 
Commuting  Filters 
Purdue  University 

Roles  of  Matched  Filtering  and  Coarse  in  Insect  Visual  Processing 
University  of  Arizona 

Prediction  of  Compression  Textures  in  Tantalum  Using  a 
Pencil-Glide  Computer  Mode  Program 
University  of  Michigan 

Parallelization  of  Time-Dependent  Maxwell  Equations  Using 
High  Perform 
University  of  Dayton 

A  Hybrid  Variational -Asymptotic  Method  for  lire  Analysis  of 
MicroMechanical  Damage  in  Composites 
Stevens  Institute  of  Technology' 

Continuous  Sliding  Mode  Control  Approach  for  Addressing 
Actuator  Deflecrion  and  Deflection  rate  Saruraton  in  Tailess 
Aircraft  Control  and  Re-Configurable  Flight  Control 
University'  of  Alabama  In  Huntsville 

Feature  Selection  for  Automatic  Target  Recognition:  Mutual 
Information  &  Statistical  Techniques 
Ohio  University 


Laboratory  & 
Directorate 

AFRL/MN 

AFRL/VASD 

AFRL/SN 

AFRL/VAAC 

AFRL/MN 

AFRL/MN 

AFRL/VASD 

AFRL/MLBM 

AFRL/VACD 

AFRL/SN 


v 


1998  SREP  Final  Technical  Report  Table  of  Contents 


Volume  5 


Principle  Report  Title 

Investigator _ University/Institution 


1  Dr.  Frank  Collins 


2  Dr.  Paul  W.  Whaley 


3  Dr.  Devendra  Kumar 


4  Dr.  Joe  G.  Chow 


Arnold  Engineering  Development  Center 

Monte  Carlo  Compution  of  Species  Separation  by  a 
Conical  Skimmer  in  Hypersonic  Transition  Flow 
University  of  Tennessee  Space  Institute 

Air  Logistics  Centers 

Probabilistic  Analysis  of  Residual  Strength  in  Corroded 
and  Uncorroded  Aging  Air  Mineralization 
Oklahoma  Christian  University  of  Science  &  Art 

Further  Development  of  a  Simpler,  Multiversion 
Control  Protocol  for  Internet  Databases 
University  of  Georgia 

An  Automated  3-D  Surface  Model  Creahon  Module  for 
Laser  Scanned  Point  Data 
Florida  International  University 


Laboratory  & 
Directorate 


AEDC 


OCALC/TIE 


SAALC 


WRALC 


vi 


1998  SUMMER  RESEARCH  EXTENSION  PROGRAM  (SREP)  MANAGEMENT  REPORT 


1.0  BACKGROUND 


Under  the  provisions  of  Air  Force  Office  of  Scientific  Research  (AFOSR)  contract  F49620-90-C- 
0076,  September  1990,  Research  &  Development  Laboratories  (RDL),  an  8(a)  contractor  in 
Culver  City,  CA,  manages  AFOSR’s  Summer  Research  Program.  This  report  is  issued  in  partial 
fulfillment  of  that  contract  (CLIN  0003AC). 

The  Summer  Research  Extension  Program  (SREP)  is  one  of  four  programs  AFOSR  manages 
under  the  Summer  Research  Program.  The  Summer  Faculty  Research  Program  (SFRP)  and  the 
Graduate  Student  Research  Program  (GSRP)  place  college-level  research  associates  in  Air  Force 
research  laboratories  around  the  United  States  for  8  to  12  weeks  of  research  with  Air  Force 
scientists.  The  High  School  Apprenticeship  Program  (HSAP)  is  the  fourth  element  of  the  Summer 
Research  Program,  allowing  promising  mathematics  and  science  students  to  spend  two  months  of 
their  summer  vacations  working  at  Air  Force  laboratories  within  commuting  distance  from  their 
homes. 

SFRP  associates  and  exceptional  GSRP  associates  are  encouraged,  at  the  end  of  their  summer 
tours,  to  write  proposals  to  extend  their  summer  research  during  the  following  calendar  year  at 
their  home  institutions.  AFOSR  provides  funds  adequate  to  pay  for  SREP  subcontracts.  In 
addition,  AFOSR  has  traditionally  provided  further  funding,  when  available,  to  pay  for  additional 
SREP  proposals,  including  those  submitted  by  associates  from  Historically  Black  Colleges  and 
Universities  (HBCUs)  and  Minority  Institutions  (Mis).  Finally,  laboratories  may  transfer  internal 
funds  to  AFOSR  to  fund  additional  SREPs.  Ultimately  the  laboratories  inform  RDL  of  their 
SREP  choices,  RDL  gets  AFOSR  approval,  and  RDL  forwards  a  subcontract  to  the  institution 
where  the  SREP  associate  is  employed.  The  subcontract  (see  Appendix  1  for  a  sample)  cites  the 
SREP  associate  as  the  principal  investigator  and  requires  submission  of  a  report  at  the  end  of  the 
subcontract  period. 

Institutions  are  encouraged  to  share  costs  of  the  SREP  research,  and  many  do  so.  The  most 
common  cost-sharing  arrangement  is  reduction  in  the  overhead,  fringes,  or  administrative  charges 
institutions  would  normally  add  on  to  the  principal  investigator’s  or  research  associate  s  labor. 
Some  institutions  also  provide  other  support  (e.g.,  computer  run  time,  administrative  assistance, 
facilities  and  equipment  or  research  assistants)  at  reduced  or  no  cost. 

When  RDL  receives  the  signed  subcontract,  we  fund  the  effort  initially  by  providing  90%  of  the 
subcontract  amount  to  the  institution  (normally  $18,000  for  a  $20,000  SREP).  When  we  receive 
the  end-of-research  report,  we  evaluate  it  administratively  and  send  a  copy  to  the  laboratory  for  a 
technical  evaluation.  When  the  laboratory  notifies  us  the  SREP  report  is  acceptable,  we  release 
the  remaining  funds  to  the  institution. 
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2.0  THE  1998  SREP  PROGRAM 


SELECTION  DATA:  A  total  of  490  faculty  members  (SFRP  Associates)  and  202  graduate 
students  (GSRP  associates)  applied  to  participate  in  the  1998  Summer  Research  Program.  From 
these  applicants  188  SFRPs  and  98  GSRPs  were  selected.  The  education  level  of  those  selected 
was  as  follows: 


1997  SRP  Associates,  by  Degree 

SFRP  1 

GSRP 

PHD 

MS 

MS 

BS 

184 

6 

2 

53 

Of  the  participants  in  the  1997  Summer  Research  Program  90  percent  of  SFRPs  and  13  percent 
of  GSRPs  submitted  proposals  for  the  SREP.  One  undred  and  thirty-two  proposals  from  SFRPs 
and  seventeen  from  GSRPs  were  selected  for  funding,  which  equates  to  a  selection  rate  of  54%  of 
the  SFRP  proposals  and  of  34%  for  GSRP  proposals. 


1998  SREP:  Proposals  Submitted  vs.  Proposals  Selected 

Summer 

1997 

Participants 

Submitted 

SREP 

Proposals 

SREPs 

Funded 

SFRP 

188 

132 

20 

GSRP 

98 

17 

4 

TOTAL 

286 

149 

24 

The  funding  was  provided  as  follows: 


Contractual  slots  funded  by  AFOSR  18 

Laboratory  funded  22 

Total  40 

Twelve  HBCU/MI  associates  from  the  1997  summer  program  submitted  SREP  proposals;  six 
were  selected  (none  were  lab-funded;  all  were  funded  by  additional  AFOSR  funds). 
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f  Proposals  Submitted  and  Selected,  by  Laboratory 

Selected 

Armstrong  Research  Site 

9 

3 

Air  Logistic  Centers 

31  ^ 

5 

Arnold  Engineering  Development  Center 

2 

1 

Phillips  Research  Site 

30  1 

10 

Rome  Research  Site 

29 

12 

Wilford  Hall  Medical  Center 

1 

0 

Wright  Research  Site 

47 

9 

TOTAL 

149 

40 

Note:  Armstrong  Research  Site  funded  1  SREP;  Phillips  Research  Site  funded  6;  Rome  Research 

Site  funded  9;  Wright  Research  Site  funded  6. 


The  125  1997  Summer  Research  Program  participants  represented  60  institutions. 


Institutions  Represented  on  the  1997  SRP  and  1998  SREP 

Number  of  schools 
represented  in  the 
Summer  97  Program 

Number  of  schools 
represented  in 
submitted  proposals 

Number  of  schools 
represented  in 
Funded  Proposals 

125 

110 

55 

Thirty  schools  had  more  than  one  participant  submitting  proposals. 


Proposals  Submitted  Per  School 


B  Submitted 
□  Selected 


Number  of  Proposals 
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The  selection  rate  for  the  65  schools  submitting  1  proposal  (68%)  was  better  than  those 
submitting  2  proposals  (61%),  3  proposals  (50%),  4  proposals  (0%)  or  5+  proposals  (25% 
The  4  schools  that  submitted  5+  proposals  accounted  for  30  (15%)  of  the  149  proposals 

submitted. 


Of  the  149  proposals  submitted,  130  offered  institution  cost  sharing.  Of  the  funded  proposals 
which  offered  cost  sharing,  the  minimum  cost  share  was  $3046.00,  the  maximum  was 
$39,261.00  with  an  average  cost  share  of  $11,069.21. 


|  Proposals  and  Institution  Cost  Sharing 

Proposals 

Proposals 

Submitted 

Funded 

With  cost  sharing 

117 

32 

Without  cost  sharing 

32 

8 

Total 

149 

40 

The  SREP  participants  were  residents  of  31  different  states.  Number  of  states  represented  at 
each  laboratory  were: 


States  Represented,  by  Proposals  Submitted/Selected  per  Laboratory 

Proposals 

Submitted 

Proposals 

Funded 

Armstrong  Laboratory 

31  1 

5  1 

Air  Logistic  Centers 

9 

3 

Arnold  Engineering  Development  Center  | 

2 

1 

Phillips  Laboratory 

30 

10 

Rome  Laboratory 

29 

- 

12 

Wilford  Hall  Medical  Center 

r  o  | 

Wright  Laboratory 

47 

9 

Nine  of  the  1997  SREP  Principal  Investigators  also  participated  in  the  1998  SREP. 


ADMINISTRATIVE  EVALUATION:  The  administrative  quality  of  the  SREP  associates’  final 
reports  was  satisfactory.  Most  complied  with  the  formatting  and  other  instructions  provided  to 
them  by  RDL  Thirty-seveb  final  reports  have  been  received  and  are  included  in  this  report. 
The  subcontracts  were  funded  by  $992,855.00  of  Air  Force  money.  Institution  cost  sharing 

totaled  $354,215.00. 
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TECHNICAL  EVALUATION:  Tbe  form  used  for  the  technical  evaluation  is  provided  as 
Appendix  2.  Thirty-five  evaluation  reports  were  received.  Participants  by  laboratory  versus 
evaluations  submitted  is  shown  below: 


Participants 

Evaluations 

Percent 

Armstrong  Laboratory 

5 

4 

95.2 

Air  Logistic  Centers 

3 

3 

100 

Arnold  Engineering  Development  Center 

1 

1 

100 

Phillips  Laboratory 

10 

10 

100 

Rome  Laboratory 

12 

12 

100 

Wright  Laboratory 

9 

5 

91.9 

Total 

40 

35 

95.0 

Notes: 

1:  Research  on  four  of  the  final  reports  was  incomplete  as  of  press  time  so  there  aren’t  any  technical 
evaluations  on  them  to  process,  yet.  Percent  complete  is  based  upon  20/21  =95.2% 

2:  One  technical  evaluation  was  not  completed  because  one  of  the  final  reports  was  incompkte  as  of 
press  time.  Percent  complete  is  based  upon  18/18  =  100% 

The  number  of  evaluations  submitted  for  the  1998  SREP  (95.0%)  shows  a  marked 
improvement  over  the  1997  SREP  submittals  (65%). 

PROGRAM  EVALUATION:  Each  laboratory  focal  point  evaluated  ten  areas  (see  Appendix 
2)  with  a  rating  from  one  (lowest)  to  five  (highest).  The  distribution  of  ratings  was  as  follows: 


Rating 

Not  Rated 

1 

2 

3 

4 

5 

#  Responses 

7 

1 

7 

62  (6%) 

226  (25%) 

617  (67%) 

The  8  low  ratings  (one  1  and  seven  2’s  )  were  for  question  5  (one  2)  “The  USAF  should 
continue  to  pursue  the  research  in  this  SREP  report”  and  question  10  (one  1  and  six  2's)  “The 
one-year  period  for  complete  SREP  research  is  about  right”,  in  addition  over  30%  of  the 
threes  (20  of  62)  were  for  question  ten.  The  average  rating  by  question  was: 
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Question 

1 

2 

3 

4 

5 

7 

8  1 

9 

10 

Average 

4.6 

4.6  1 

msm 

msm 

4.6 

msm 

4.8 

4.5 

4.6 

4.0 

The  distribution  of  the  averages  was: 


Area  10  “the  one-year  period  for  complete  SREP  research  is  about  right  had  the  lowest 
average  rating  (4.1).  The  overall  average  across  all  factors  was  4.6  with  a  small  sample 
standard  deviation  of  0.2.  The  average  rating  for  area  10  (4.1)  is  approximately  three  sigma 
lower  than  the  overall  average  (4.6)  indicating  that  a  significant  number  of  the  evaluators  feel 
that  a  period  of  other  than  one  year  should  be  available  for  complete  SREP  research. 
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The  average  ratings  ranged  from  3.4  to  5.0.  The  overall  average  for  those  reports  that  were 
evaluated  was  4.6.  Since  the  distribution  of  the  ratings  is  not  a  normal  distribution  the  average 
of  4.6  is  misleading.  In  fact  over  half  of  the  reports  received  an  average  rating  of  4.8  or 
higher.  The  distribution  of  the  average  report  ratings  is  as  shown: 


It  is  clear  from  the  high  ratings  that  the  laboratories  place  a  high  value  on  AFOSR’s  Summer 
Research  Extension  Programs. 
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3.0  SUBCONTRACTS  SUMMARY 


Table  1  provides  a  summary  of  the  SREP  subcontracts.  The  individual  reports  are  published 
volumes  as  shown: 

Laboratory  Volume 

Armstrong  Research  Site  1 

Arnold  Engineering  Development  Center  5 

Air  Logistic  Centers  5 

Phillips  Research  Site  2 

Rome  Research  Site  3 

Wright  Research  Site  4 
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SREP  SUB-CONTRACT  DATA 


Report  Author  Author's 

Author's  University  _  Ptgrtf 

Chubb  ,  Geradd  "  PhD~ 

Industrial  Engineering  98-0829 


Ohio  SUte  University,  Columbus,  OH 


Sponsoring 

Lab 

AIVHR 

Scoring 


Contract 

Performance  Period  Amount 

01/01/98  12/31/98  $25000.00 

Pilot  Performance  of  Basic 
Manuevers 


Univ.  Cost 
Share 


S0.00 

Flight 


Foy ,  Brent  PhD 

Medical  Physics  38-0828 

Wright  State  University,  Dayton,  OH 


AL/OE  01/01/98  12/31/98  $25000.00  $11278.00 

Development  fc  Validation  of  a 
Physio logically -Based  Kinetic  Model  ci  Perfused 


Lance ,  Charles  PhD 

Psychology  98-0842 

Univ  of  Georgia  Res  Foundation,  Athens,  GA 


AIVHR  01/01/98  12/31/98  $24989.00  $0.00 

Extension  of  Job  Performance  Measurement  Tech  tc 
the  Development  of  a  Prototype 


Woehr ,  David  PhD 

Department  of  Psychology  98-08  02 

Texas  A  &  M  Univ-College  Station,  College 


AIVHR  01/01/98  12/31/98  $25000.00  $11508.00 

Validation  cf  The  Multidimensional  werk  ethic 
profile  (MWEF)  as  a  screening  too 


Collins ,  Frank  Phl> 

Mechanical  Engineering  98-0807 

Tennessee  Univ  Space  Institute,  Tullahoma,  TN 


AEDC/E  01/01/98  12/31/98  $25000.00  $16104.00 

Merit e  Carlo  Computation  of  SpeciesSepaeration  by 
a  Conical  Skimmer  in  Hypersonic 


Whaley ,  Paul  PhD 

Mechanical  Engineering  98-0820 

Oklahoma  Christian  Univ  of  Science  &  Art, 


ALC/OC  01/01/98  12/31/98  $23351.00  $3046.00 

Probabilistic  Analysis  of  Residual  Strength  in 
Corroded  and  Dncorroded  Aging  Air 


Ball* ,  Mark  PhD 

Applied  Math  98-0816 

Univ  of  Colorado  at  Boulder,  Boulder,  CO 


PL/SX  01/01/98-  12/31/98  $25000.00  $0.00 

Non-Linear  Adaptive  Control  for  a  Precision 
Deployable  Structure  with  White  ligh 


Dune,  Neb  PM> 

Astrophysics  98-08  08 

University  of  New  Mexico,  Albuquerque,  NM 

Hanson ,  George  PhD 

Electrical  Engineering  98-0811 

Univ  of  Wisconsin  -  Milwaukee,  Milwaukee,  W1 


PL/Ll  01/01/98  12/31/98  $25000.00  $5777.00 

Image  Recovery  Using  Phase  Diversity 

PL/WS  01/01/98  12/31/98  $25000.  M  $23250.00 

Perturbation  Analysis  of  the  Natural  Frequencies 
Targets  in  Inhomogeneous  Media 


Jeffs ,  Brian  PhD 

Electrical  Engineering  98-0813 

Brigham  Young  University,  Provo,  CT 

Kar  ,  Aravin  da  PhD 

Engineering  98-0812 

University  of  Central  Florida,  Orlando,  FL 

Leo  ,  Donald  PhD 

Mechanical  &  Aerospace  98-08IC 

University  of  Toledo,  Toledo,  OH 

Liu  ,  Hanli  PhD 

Physics  98-0814 


Univ  of  Texas  at  .Arlington,  Arlington,  TX 


PL/U  01/01/98  12/31/98  S25000.t0  $19177.00 

Bayesian  Restoration  of  Space  object  Images  From 
Adaptive  Optics  Data  with  urJcr.o 

PL/LI  01/01/98  12/31/98  $25000.00  $5414.00 

Effects  of  Vapor-Plasma  Layer  on  Thirx- Section 
Cutting  and  Calculation  of  Modes 

PLVVT  01/01/98  #9/30/98  $24964.  #0  $9628.00 

Adaptive  vibration  suppression  for  autonomous 
Control  Systems 

PL/LI  01/01/98  12/31/98  $25000.00  $11000.00 

Continuous -Wave  Approach  to  3-D  Imaging  through 
Turbid  media  w/a  Single  Planar  M 


Bienfang ,  Joshua 

Physics  98-0815 

University  of  New  Mexico,  Albuquerque,  NM 

Paulson  ,  Eric  BS 

Engineering/Physics  98-08  37 

Univ  of  Colorado  at  Boulder,  Boulder,  CO 


PL/LI  01/01/98  12/31/98  S24994.W  $0.00 

Optical  Clocks  Based  on  Diode  Lasers 

PURK  01/01/98  12/31/98  $25000.00  57794.00 

Optimization  &  Analysis  of  a  Wave n her  Vehicle 
for  Global  Spacepiane  Tra jeer :rie 


SREP  SUB-CONTRACT  DATA 


Report  Author 
Author's  University 


Stephens  D ,  Kenneth 


Author's  Sponsoring  n  .  D  .  .  uoiitraet  tniv.  cos 

Degree  Lib  Performance  Penod  share 


Contract 

Amount 


Univ.  Cost 
Share 


MA  PLWS  01/0L*98  12/31/98  $25000.00  $16764.00 

98-0909  Simulation  of  an  Explosively  Formed  Fuse  Using 


University  *f  North  Texas,  Denton,  TX 

Barjaktarovic ,  Milica 
Electrical  Engineering 
Wilkes  University,  Wilkes  Barrt,  PA 

Batalama ,  Stella 
EE 

SUNY  Buffalo,  Buffalo,  NY 


MACH  2 

PhD  RL/IW  01/01/98  12/31/98  $24976.00  $3158.00 

98-0824  Specification  and  Verification  of  SDN.7C1  MSP 
Functions  and  Missi  Crypto  Functic 

PhD  RL/C3  01/01/98  12/31798  $25000.00  $5600.00 

98-082  3  Robust  Spread  Spectrum  Communi  cat  ions :  Adaptive 
Interference  Mitigation  Technique 


Bourtakis ,  Nikolaos  PhD 

Computer  Science  &  Engr  98-0832 

SUNY  Binghamton,  Binghamton,  NY 

Dasigi ,  Vesugopala  PhD 

Computer  Science  98-0830 

Southern  Polytechnic  State  Univ,  Marietta,  GA 

Eckert ,  Richard  PhD 

Physic $  98-0825 

SUNY  Binghamton,  Binghamton,  NY 

Lin ,  Kuo-Chi  PhD 

Aerospace  Engineering  98-0822 

University  of  Central  Florida,  Orlando,  FL 

Pados ,  Dimitrios  PhD 

Dept  of  Electrical  /Computer  Eng.  98-0818 
State  Unh*.  of  New  York  Buffalo,  Buffalo,  NY 


RLTR  01/01/98  12/31/98  $25000.00  $22723.00 

hierarchical -Adaptive  Image  Segmentation 

RL/C3  01/01/98  12/31/98  $25000.00  $4000.00 

Information  Fusion  w/Multiple  Feature  Extractors 
for  automatic  Text  Classificati 

R1VC3  01/01/98  12/31/98  $25000.00  $39261.00 

The  Interactive  Learning  Wall;  A  PC-Based, 
Deployable  Data  Wall  for  Use  in  a  Co 

RLTR  01/01/98  12/31/98  $25000.00  $0.00 

Web-Based  Distributed  Simulation 


RJL/OC  01/01/98  12/31/98  $25000.00  $5600.00 

Adaptive  Array  Radars  and  Joint  Space-Time 
Auxiliary  Verctor  Filtering 


Panda ,  Brajendra  PhD  R1VCA  01/01/98  12/31/98  $25000.00  $7113.00 

Computer  Science  98-0821  Information  Warfare:  Design  of  an  Efficient  Log 

University  of  North  Dakota,  Grand  Forks,  ND  Management  Method  to  Aid  In  Dat 


Pittarelli ,  Michael  PhD 

Systems  Science  98-0827 

SUNY  OF  Tech  Utica,  Utica,  NY 

Schmalz ,  Mark  PhD 

Dept  of  Computer  &  Info  Science  93-C931 

University  of  Florida,  Gainesville,  FL 

Ye ,  Nong  PhD 

Industrial  Engineering  98-C826 

Arizona  State  University,  Tempe,  AZ 

Bradley ,  Parker  BS 

Physics  98-0834 

Syracuse  University,  Syracuse,  NY 

Kumar ,  Devendra  PhD 

Computer  Science  98-0805 

CUNY-City  College,  New  York,  NY 

Chow ,  Joe  PhD 

Mechanical  Engineering  98-0806 

Florida  International  Univ,  Miami,  FL 


RL/C3  01/01/98  12/31/98  $24998.00  $0.00 

Complexity  of  Detecting  and  content -driven 
methods  for  resolving  database  inccrs 

RL/IR  01/01/98  12/31/98  $24619.00  $0.00 

Errors  Inherent  in  3D  Target  Reconstruction  from 
Multiple  Airborne  Images 

RIVCA  01/01/98  12/31/98  $25000.00  $5000.00 

Model-Based  Assessment  of  Campaign  ? Ian 
Performance  under  Uncertainty 

RLTR  01/01/98  12/31/98  $25000.00  $0.00 

Development  of  User-Friendly  Comp  Environment 
for  Blind  Source  Separation  Studie 

ALC/SA  01/01/98  12/31/98  $25000.00  SI  1362.00 

Further  Development  of  a  Simpler,  Multiversion 
concurrency  Control  Protocol  for 

ALC/W  01/01/98  12/31/98  $25000.00  $5360.00 

An  Automated  3-D  Surface  Model  Creaticr.  Module 
for  Laser  Scanned  Point  Data 


SREP  SUB-CONTRACT  DATA 


Report  Author  Author’s 

Author’s  University  _ 

Beecken  ,  Brian  P*H) 

Physics  98  -C  9  04 

Bethel  College,  St  Paul,  MN 

Beggs ,  John  PU) 

Electrical  Engineering  98-0817 

Mississippi  Sute  University.  Mississippi  Sttte, 

Bhatnagar ,  Raj 

Computer  Science  98-0819 

University  of  Cincinnati,  Cincinnati,  OH 

Blaisdefl ,  Gregory 

Mechanical  Engineering  98-0839 

Purdue  University,  West  Lafayette,  IN 

Douglass ,  John  PbB 

Zoology  98-0803 

University  of  Arizona,  Tucson,  AZ 

Hosford ,  William  PHD 

Mettahirgy  98-0840 

Univ  of  Michigan,  Ann  Arbor,  M3 

Pan  ,  Yi 

Computer  Science  98-08  38 

University  of  Dayton,  Dayton,  OH 

Pochiraju ,  Kishore  PU) 

Mechaakal  Engineering  98-0833 

Stevens  Inst  of  Technology  Hoboken,  NJ 

Shtessd  ,  Yuri  PW> 

Electrical  Engineering  98-0841 

Univ  of  Alabama  at  Huatrrille,  Huntsville,  AL 

Starzyk ,  Janusz  PW) 

Electrical  Engineering  98-0801 

Ohio  University,  Athens.  OH 


Contract  Univ.  Cost 

s  Performance  Period  .Amount  Share 

WL/MN  01/01/98  12/31/98  S19986.00  $3997.00 

Development  of  a  statistical  *odel  predicting 
the  impact  cf  a  scene  projector's 

WIVFI  01/01/98  12/31/98  S25000.00  S 25 174.00 

Implementation  of  an  Optimization  Algorithm  in 
Electromagnentics  for  Fadar  Absor 

WIVAA  01/01/98  09/30/98  525000.00  SI 7488.00 

Analysis  of  Intra-Class  Variability  &  synthetic 
Target  Models  for  Use  in  ATR 

WL/FI  01/01/98  12/31/98  $25000.00  SI  1844.00 

Validation  of  a  Large  Eddy  Simulation  Code  & 
Development  of  Commuting  Filters 

WL/MN  01/01/98  12/31/98  525000.00  S3719.00 

Roles  of  Matched  Filtering  and  Coarse  in  Insect 
Visual  Processing 

WL/MN  01/01/98  12/31/98  S25000.00  S5000.00 

Prediction  of  Compression  Textures  in  Tantalum 
Using  a  Pencil -Glide  Computer  Mod 

WL/FI  01/01/98  12/31/98  $25000.00  S9486.00 

Parallelization  of  Time -Dependent  Maxwell 
Equations  Using  High  Perform.  Fortran 

WL/ML  01/01/98  12/3198  $25000.00  S9625.00 

A  Hybrid  Variational -Asymptotic  Method  for  the 
Analysis  of  MicroMechanical  Damag 

WL/FI  01/0198  12/31/98  $25000.00  S4969.00 

Continuous  Sliding  Mode  Control  Approach  for 
Addressing  actutor  Deflection  and 

WL/AA  01/01/98  12/31/98  $24978.00  S  12996.00 

Feature  Selection  for  A_tomatic  Target 
Recognition: Mutual  Info  &  Stat  Tech 
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APPENDIX  1: 


SAMPLE  SREP  SUBCONTRACT 


AIR  FORCE  OFFICE  OF  SCIENTIFIC  RESEARCH 
1998  SUMMER  RESEARCH  EXTENSION  PROGRAM 
SUBCONTRACT  98-0812 

BETWEEN 


Research  &  Development  Laboratories 
5800  Uplander  Way 
Culver  City,  CA  90230-6608 


AND 


University  of  Central  Florida 
Office  of  Sponsored  Research/  Admin#423 
4000  Central  Florida  Blvd. 
Orlando,  FL  32816-0150 


REFERENCE:  Summer  Research  Extension  Program  Proposal  97-0018 

Start  Date:  01/01/98  End  Date  12/3 1/98 

Proposal  Amount:  S25000.0 
Proposal  Title. 

Effects  of  Vapor-Plasma  Layer  on  Thick-Section  Cutting  and  Calculation  of 

Modes 


(1)  PRINCIPAL  INVESTIGATOR: 

DR  Aravinda  Kar 
CREOL 

University  of  Central  Florida 
Orlando,  FL  32816-2700 

(2)  UNITED  STATES  AFOSR  CONTRACT  NUMBER:  F49620-93-C-0063 

(3)  CATALOG  OF  FEDERAL  DOMESTIC  ASSISTANCE  NUMBER  (CFDA):  12.800 
PROJECT  TITLE:  AIR  FORCE  DEFENCE  RESEARCH  SOURCES  PROGRAM 

(4)  ATTACHMENTS 

1  REPORT  OF  INVENTIONS  AND  SUBCONTRACT 

2  CONTRACT  CLAUSES 

3  FINAL  REPORT  INSTRUCTIONS 


*«*  SIGN  SREP  SUBCONTRACT  AND  RETURN  TO  RDL  *** 
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nArynworND:  Research  &  Development  Laboratories  (RDL)  is  under  co 
iF 49620-93 -C-0063 )  to  the  United  States  Air  Force  to  administer  the  Summ 
Research  Program  (SRP).  sponsored  by  the  Air  Force  Office  of  Scientific  Research 
,  AFOSR),  Bolling  Air  Force  Base,  D  C  Under  the  SRP,  a  selected  number  of  college 


faculty  members  and  ^aduate  students  spend  part  of  the  summer  conducting  research 
Air  Force  laboratories.  After  completion  of  the  summer  tour  participants  may 
submit,  through  their  home  institutions  proposals  for  follow-on  research.  Fhe  follow- 


on  research  is  known  as  the  Summer  Research  Extension  Program  (SREP). 
Approximately  61  SREP  proposals  annually  will  be  selected  by  the  Air  Force  for 
funding  of  up  to  $25,000;  shared  funding  by  the  academic  institution  is  encouraged. 
SREP  efforts  selected  for  Binding  are  administered  by  RDL  through  subcontracts  with 
the  institutions.  This  subcontract  represents  an  agreement  between  RDL  and  the 
institution  herein  designated  in  Section  5  below. 


2  ptm  PAYMENTS:  RDL  will  provide  the  following  payments  to  SREP  institutions. 

.  80  percent  of  the  negotiated  SREP  dollar  amount  at  the  start  of  the  SREP 

research  period. 

.  Tne  remainder  of  the  funds  within  30  days  after  receipt  at  RDL  of  the 
acceptable  written  final  report  for  the  SREP  research. 


INSTITUTION’S  RESPONSIBILITIES: 


As  a  subcontractor  to  RDL,  the  institution 


designated  on  the  title  page  will. 


a.  Assure  that  the  research  performed  and  the  resources  utilized  adhere  to  those 
defined  in  the  SREP  proposal. 

b.  Provide  the  level  and  amounts  of  institutional  support  specified  in  the  SREP 
proposal.. 

c.  Notify  RDL  as  soon  as  possible,  but  not  later  than  30  days,  of  any  changes  in 
3  a  or  3  b  above,  or  any  change  to  the  assignment  or  amount  of  participation  of 
the  Principal  Investigator  designated  on  the  title  page 

d.  .Assure  that  the  research  is  completed  and  the  final  report  is  delisered  to  RDL 
not  later  than  twelve  months  from  the  effective  date  of  this  subcontract,  but  no 
later  than  December  31.  1998.  The  effective  date  of  the  subcontract  is  one 
week  after  the  date  that  the  institution’s  contracting  representative  signs  this 
subcontract,  but  no  later  than  January  15,  1998. 

e.  Assure  that  the  final  report  is  submitted  in  accordance  with  Attachment  3. 

f.  Agree  that  any  release  of  information  relating  to  this  subcontract  (news 
releases,  articles,  manuscripts,  brochures,  advertisements,  still  and  motion 
pictures,  speeches,  trade  associations  meetings,  symposia,  etc.)  will  include  a 
statement  that  the  project  or  effort  depicted  was  or  is  sponsored  by:  Air  Force 
Office  of  Scientific  Research,  Bolling  AFB,  D  C. 

g.  Notify  RDL  of  inventions  or  patents  claimed  as  the  result  of  this  research  as 
specified  in  Attachment  1 . 

h.  RDL  is  required  by  the  prime  contract  to  flow  down  patent  rights  and  technical 
data  requirements  to  this  subcontract.  Attachment  2  to  this  subcontract 
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contains  a  list  of  contract  clauses  incorporated  by  reference  in  the  prune 


contract. 


4.  All  notices  to  RDL  shall  be  addressed  to. 


RDL  AFOSR  Program  Office 
5800  Uplander  Way- 
Culver  City,  CA  90230-6609 

By  their  signatures  below,  the  parties  agree  to  provisions  of  this  subcontract. 


Abe  Sopher 

RDL  Contracts  Manager 


Sienature  of  Institution  Contracting  Official 


Typed/Printed  Name 


Date 


Title 


Institution 


Date/Phone 
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ATTACHMENT  2 
CONTRACT  CLAUSES 


This  contract  incorporates  by  reference  the  following  clauses  of  the  Federal  Acquar-on 
Regulations  (FAR),  with  the  same  force  and  effect  as  if  they  were  giver .in  ffiU  texL  Upon 
request,  the  Contracting  Officer  or  RDL  will  make  their  full  text  available  (FAR  52  .52  2) 


FAR  CLAUSES 


TTTT.E  AND  DATE 


52.202- 1 

52.203- 3 

52.203- 5 

52.203- 6 


DEFINITIONS 

GRATUITIES 

COVENANT  AGAINST  CONTINGENT  FEES 

RESTRICTIONS  ON  SUBCONTRACTOR 
SALES  TO  THE  GOVERNMENT 


52.203- 7 

52.203- 8 


52.203-10 


52.203-12 


ANTI-KICKBACK  PROCEDURES 

CANCELLATION,  RECISSION,  AND 

RECOVERY  OF  FUNDS  FOR  ILLEGAL  OR  IMPROPER 

ACTIVITY 

PRICE  OR  FEE  ADJUSTMENT  FOR  ILLEGAL 
OR  IMPROPER  ACTIVITY 

LIMITATION  ON  PAYMENTS  TO  INFLUENCE 
CERTAIN  FEDERAL  TRANSACTIONS 


52.204-2 


52.209-6 


52.212-8 

52.215- 2 

52.215- 10 


SECURITY  REQUIREMENTS 

PROTECTING  THE  GOVERNMENT’S 
INTEREST  WHEN  SUBCONTRACTING  WITH 
CONTRACTORS  DEBARRED,  SUSPENDED,  OR 
PROPOSED  FOR  DEBARMENT 

DEFENSE  PRIORITY  AND  ALLOCATION 
REQUIREMENTS 

AUDIT  AND  RECORDS  -  NEGOTIATION 

PRICE  REDUCTION  FOR  DEFECTIVE  COST 
OR  PRICING  DATA 


I;  astdalA  wp  srrp  Atlcfa_2.doc 
Rev.  I?* 
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52.215-12 

SUBCONTRACTOR  COST  OR  PRICING 

DATA 

52.215-14 

INTEGRITY  OF  UNIT  PRICES 

52.215-8 

ORDER  OF  PRECEDENCE 

52.215.18 

REVERSION  OR  ADJUSTMENT  OF  PLANS 

FOR  POSTRETIREMENT  BENEFITS  OTHER 
THAN  PENSIONS 

52.222-3 

CONVICT  LABOR 

52.222-26 

EQUAL  OPPORTUNITY 

52.222-35 

AFFIRMATIVE  ACTION  FOR  SPECIAL 
DISABLED  AND  VIETNAM  ERA 

VETERANS 

52.222-36 

AFFIRMATIVE  ACTION  FOR 

handicapped  workers 

52.222-37 

EMPLOYMENT  REPORTS  ON  SPECIAL 
DISABLED  VETERAN  AND  VETERANS  OF  THE 
VIETNAM  ERA 

52.223-2 

CLEAN  AIR  AND  WATER 

52.223-6 

DRUG-FREE  WORKPLACE 

52.224-1 

PRIVACY  ACT  NOTIFICATION 

52.224-2 

PRIVACY  ACT 

52.225-13 

RESTRICTIONS  ON  CONTRACTING  WITH 
SANCTIONED  PERSONS 

52.227-1 

ALT.  I  -  AUTHORIZATION  AND  CONSENT 

52.227-2 

NOTICE  AND  ASSISTANCE  REGARDING 
PATIENT  AND  COPYRIGHT  INFRINGEMENT 

I;  -osrdii*'  wpsrcpattch_2.doc 

Rev. 


fervtfwctta o-l* 


52.227- 10 

52.22"’- 1 1 

52.228- 7 


52.230-5 


52.232-23 


52.233-1 


52.233-3 


52.237-3 


52.246-25 


52.247-63 

52.249-5 


52.249-14 


52.251-1 


FILING  OF  PATENT  APPLICATIONS  - 
CLASSIFIED  SUBJECT  MATTER 

PATENT  RIGHTS  -  RETENTION  BY  THE 
CONTRACTOR  (SHORT  FORM) 
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Abstract 

In  this  document  we  continue  to  demonstrate  the  process  of  modular  design  and  step-wise  system 
refinement.  We  use  the  case  study  of  a  real-life  system,  the  Multilevel  Information  System  Security 
Initiative  (MISSI).  MISSI  is  a  National  Security  Agency  (NSA)  program,  designed  to  send  protected 
messages  over  unprotected  networks  such  as  Internet.  MISSI  employs  cryptography  by  using  a 
credit  card  sized  Personal  Computer  Memory  card  Interface  Association  (PCMCIA)  card  called  the 
FORTEZZA  Crypto  Card. 

MISSI  relies  on  X.500.  X.400.  SDN.701-901  standards,  as  well  as  FORTEZZA  documentation. 
Our  research  so  far  was  focusing  on  specifying  various  parts  of  MISSI  sender  and  receiver,  in  order 
to  demonstrate  modular  design  and  stepwise  refinement.  We  formally  defined  various  requrements 
either  directly  specified  or  indirectly  implied  in  the  standards  and  documents  describing  MISSI,  such 
as:  message  structure.  MISSI  access  and  forwarding  policy,  sender,  and  local  cache.  We  assumed 
that  the  FORTEZZA  card  implements  the  necessary  steps  to  carry  out  the  given  commands.  In 
this  report  we  take  the  specification  into  one  more  level  of  detail:  we  specify  the  interface  between 
FORTEZZA  card  and  application  software,  and  the  steps  to  be  taken  by  FORTEZZA  card.  We  used 
formal  language  called  PROMELA,  based  on  Hoare’s  CSP.  We  verified  the  model  using  automated 
model  checker  SPIN,  developed  by  AT&T. 
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Chapter  1 


Introduction 


In  the  process  of  software  design  and  testing,  it  is  practical  to  rely  on  modular  design  and  step-wise 
refinement  in  order  to  streamline  and  manage  the  process,  especially  if  the  system  is  secure  and/or 
critical.  We  have  used  a  real-life  complex  system  to  investigate  how  formal  methods  can  assist  in 
software  design  and  testing. 

Our  case  study  is  the  Multilevel  Information  System  Security  Initiative  (MISSI),  a  National 
Security  Agency  (NSA)  program,  designed  to  send  protected  messages  over  unprotected  networks 
such  as  Internet  [2].  MISSI  relies  on  cryptography  by  using  a  credit  card  sized  Personal  Computer 
Memory  card  Interface  Association  (PCMCIA)  card  called  the  FORTEZZA  Crypto  Card  (or  ''Card," 
for  short). 

FORTEZZA  Card  is  not  specific  to  MISSI  system.  As  such,  the  Card  is  designed  to  run  with  any 
application  which  requires  security  services.  The  cryptographic  hardware  on  the  Card  is  accessed 
via  high-level  C  functions,  called  the  Cl  library  of  functions. 

On  the  user's  side.  MISSI  relies  on  SDN  standards  to  invoke  the  proper  security  services.  In  other 
words,  there  is  a  software  on  the  user’s  workstation,  called  User  Agent  (UA),  which  translates  user 
requests  into  appropriate  action.  UA  invokes  high-level  functions  called  MSP  functions,  according 
to  the  SDN  standard. 

Since  FORTEZZA  and  SDN  standards  are  two  separate  standards,  we  must  be  concerned  with 
their  interface.  All  ambiguities  must  be  cleared.  In  particular,  are  there  any  omissions  and/or  errors 
which  lead  to  security  holes?  For  example,  what  happens  when  a  user  removes  and  inserts  the 
card(s)? 

From  the  engineering  perspective,  the  two  basic  questions  are: 

•  how  do  we  design  and  build  the  interface? 

•  how  do  we  know  that  it  works  correctly? 

In  this  report,  we  will  discuss  the  above  questions. 

This  report  is  a  continuation  of  our  work  presented  in  [3].  The  goals  of  our  work  are: 
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•  help  the  designers  of  complex  systems  such  as  MISSI  comprehend  and 
design  :he  system  in  an  organized  way 

•  help  the  communication  between  teams  designing  and  building  various 
parts  c:  the  system 

•  eventually  hand  the  verified  formal  specification  to  the  programmers  and 
builders:  based  on  their  input,  change  the  specification,  and  repat  this 
step. 

We  chose  formal  language  PROMELA.  based  on  Hoare’s  CSP,  to  specify  and  test  MISSI  receiving 
and  forwarding. 

Chapter  2  gives  an  overview  of  the  formal  tool  we  used,  SPIN;  Chapter  3  describes  the  MISSI  ap¬ 
plication;  Chapter  4  presents  a  detailed  description  of  Cl  and  MSP  functions;  Chapter  5  presents  an 
example  of  research  done  so  far  and  the  results:  and  Chapter  8  presents  the  conclusion  and  discusses 
the  future  research.  We  have  also  included  a  brief  discussion  of  cross-certificates  in  Chapter  7. 
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Chapter  2 

Promela  and  SPIN 

Promela  is  a  language  based  on  CSP  and  implemented  in  C.  SPIN  is  the  publicly  distributed  auto¬ 
mated  verification  tool  which  supports  Promela  [6,  8,  i,  9]. 

Promela  uses  processes  (we  can  think  of  them  as  “C  functions  executed  concurrently.” )  Processes 
are  executed  by  being  called  from  another  process  using  the  run  statement.  The  main  process  is 
called  init. 

Promela  uses  channels  for  communication.  A  channel  is  a  FIFO  queue  of  a  certain  capacity.  For 
example,  a  channel  which  can  store  N  byte-long  messages  is  declared  as: 

chan  channel  =  [N]  of  {byte} 

Zero-capacity  channels  do  not  have  capability  to  store  messages,  and  are  used  for  synchronization 
(i.e.  handshaking.)  The  following  syntax  is  used  to  append  a  message  type  msg.type  with  values 
exprl  and  erpr2  to  channel  q:  q!nsg_type(exprl ,  expr2).  q?msg  means  "receive  message  msg  on 
channel  q.” 

Promela  has  twelve  types  of  statements:  assertion,  expression,  selection,  assignment,  goto,  break, 
repetition,  atomic,  send,  receive,  printf,  and  timeout,  skip  statement  is  analogous  to  C  continue 
statement.  Assertions,  expressions  and  printf  statements  are  similar  to  C. 

Selection  statement  is  similar  to  C  i if  construct,  except  that  Promela  requires  that  one  option 
be  executed,  and  will  deadlock  if  no  option  is  executable.  Repetition  statement  is  similar  to  C  do 
loop.  It  is  executed  repeatedly  until  a  break  statement  is  executed  or  a  goto  statement  transfers 
control  outside  of  the  loop.  Atomic  sequence  is  indicated  by  the  keyword  atomic  and  allows  a  block 
of  critical  code  to  be  executed  without  interruption. 

SPIN  can  use  assert  statements,  never  claims  and  Linear  Temporal  Logic  (LTL)  statements  for 
validation.  Assert  statements  state  properties  of  data  at  a  given  state.  Never  claims  state  properties 
of  data  at  one  or  more  states. 

An  atomic  statement  marks  the  specification  section  that  is  to  be  executed  without  interruptions: 
i.e.  it  is  a  semaphore  protecting  critical  code,  and  is  used  for  streamlining  the  model. 
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Chapter  3 

Overview  of  MISSI  Workstation 


The  overview  of  MISSI  system  is  given  in  [5].  In  this  document  we  will  give  a  brief  overview  of  the 
MISSI  workstation. 

Each  MISSI  workstation  may  be  connected  to  a  FORTEZZA  card  reader.  Each  user  authorized 
to  use  card  reader  services  posses  a  FORTEZZA  card.  Cards  are  used  for  security  functions,  such  as 
encryption  and  digital  signatures.  The  Card  is  contacted  via  SDN. 801  Message  Security  protocol. 
The  process  of  sending  out  a  message  to  the  Internet  involves  using  X.400  or  RFC  1521  message 
formatting  mechanisms.  X.400  is  a  suite  of  standards  for  message  formatting  and  transfer.  Each 
workstation  has  application  software  called  User  Agent  (UA).  which  sends  and  receives  messages. 

Encryption  and  signatures  require  public  and  secret  keys.  A  certificate  is  a  data  structure  which 
includes  each  user’s  name  and  public  keys,  signed  by  the  authority  that  issued  the  certificate.  X.509 
standard  specifies  the  certification  hierarchy  and  certificate  and  key  management.  Certificates  are 
kept  in  a  public  depository  called  the  Directory,  specified  by  X.500  standard. 

Workstations  are  equipped  with  either  FORTEZZA  Cards  (F)  or  FORTEZZA  Plus  cards  (FP), 
or  they  have  no  cards.  FP  cards  handle  Secret  messages.  F  cards  handle  Sensitive  But  Unclassified 
messages.  Users  are  certified  by  the  Certification  Authority  (CA)  operating  at  the  CA  workstation 
(CAW). 

The  abstract  view  of  the  interface  between  a  MISSI  workstation  and  FORTEZZA  card  consists 
of  three  interacting  units:  UA,  MSP  functions,  and  the  Card.  UA  application  process  contacts  the 
hardware  on  the  Card  through  MSP  application  software.  MSP  application  software  represents  a 
high-level  interface  to  the  Card  and  consists  of  a  library  of  eight  msp_  functions.  These  functions  call 
on  the  library  cf  fifty  one  Crypto  Interface  (Cl)  CI_  functions,  which  interface  with  the  card  device 
driver  and  eventually  with  the  hardware.  A  card  is  inserted  into  a  slot  in  a  card  reader.  Each  slot 
has  a  socket  associated  with  it. 

The  interfaces  are  shown  in  Fig.  3.1.  In  Figure  3.2  we  show  the  software  components  within  a 
MISSI  workstation  and  its  connection  to  the  FORTEZZA  hardware.  The  details  of  the  relationships 
between  Card.  Library,  and  sockets  are  outlined  in  section  4.1. 
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User  Agent  /  Security  protocol  Security  protocol  /  Crypto  card  interface 

interface  (PCMCIA  interface  spec  2.1., 

(DMS  supplied  MSP  interface)  DMS  supplied  FORTEZZA  Interface  document) 

i.e.  8  msp_  functions  i.e.  5  1  CI_  functions 


Figure  3.1:  Abstract  view  of  MISSI  workstation  /FORTEZZA  card  interface. 


Overview  of  MISSI  Architecture 


Figure  3.2:  Local  workstation  components. 
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Chapter  4 

MSP  and  Cl  Functions 

In  this  chapter  we  will  refer  to  Fig. 3.2  and  section  3. 

msp-  functions  are  a  high-level  interface  to  Cl.  functions.  Cl.  functions  are  grouped  into  a  Cl 
library,  and  executed  using  a  Card.  Card  is  inserted  into  a  slot  operated  by  a  socket. 

4.1  Definitions 

We  must  more  formally  define  various  terminologies  scattered  throughout  the  MSP  an:  Cl  docu- 
mentation. 

A  Card  is  physically  inserted  into  a  socket.  The  socket  has  to  be  opened  and  select* i  in  order 
for  functions  to  be  applied  to  the  Card. 

Each  card  is  unlocked  by  only  one  PIN.  which  can  be  either  User  or  SSO  type.  SSO-type 
PINs  belong  to  SSO-enabled  users,  who  have  the  power  to  load  and  unload  information  :n  a  card. 
User-type  PINs  are  used  to  log  in  users  who  do  not  have  SSO  privileges. 

Each  Card  stores  certificates.  The  certificates  ordinarily  belong  to  several  personalities  of  the 
user  who  owns  the  PIN.  It  would  be  possible  that  several  users  know  the  PIN  that  unlorxs  a  given 

card. 

Each  user  supplies  a  PIN  to  a  card,  and  eventually  is  prompted  to  choose  one  of  the  several 
personalities  residing  on  the  Card.  Personalities  can  be  changed  any  time  during  a  sesskn.  without 
having  to  log  out. 

To  log  in  a  User,  in  case  of  non-SSO  users,  is  accomplished  by  using  mspJLogin.  Logging  in 
means  to  supply  a  PIN  that  unlocks  the  card.  In  this  case,  the  Card  is  reset  (explained  below). 
The  security  functions  on  the  Card  are  not  available  yet.  until  the  user  select  a  personality,  which 
is  beyond  the  domain  of  msp -login. 

To  reset  a  Card  means  to  “zeroize  all  registers  and  common  memory,  and  to  log  :ff  User  or 
SSO  Enabled  User”  [10].  [1]  p.25  defines  reset  as  “clearing  all  internal  registers.” 

To  log  off  a  User  is  not  explicitly  specified.  For  non-SSO  users,  we  assumed  that  it  means 
to  change  the  state  of  Card  to  Userlnitialized,  so  that  the  User  must  supply  his  PIN  again  and  go 


MO 


through  the  logging  on  process,  using  msp_login;  i.e.  once  a  user  logs  in,  he  stays  logged  in  until 
he  removes  the  card  or  logs  out.  The  meaning  of  logging  off  for  a  SSG  enabled  user  is  not  explicitly 
specified  either,  and  is  not  clear.  This  would  be  the  topic  of  additional  research. 

Informally,  the  login  algorithm  is  as  follows  (assuming  that  PIN  is  correct): 

user  requests  login 
MSP  UA  calls  on  msp.login 

a  socket  is  opened  or  selected,  card  is  inserted 
Card  is  Reset,  Card  State  =  User  Initialized 
user  supplies  PIN  => 

card  is  unlocked,  Card  State  =  Standby 

After  login,  the  user  is  obliged  to  select  a  personality  using  Cl-Select  in  order  to  achieve  the 
card  state  Ready. 

More  formally,  msp JLogin  algorithm  is  as  follows: 

CI_Initialize  /^initialize  Cl  library*/ 

CI_Reset  /*reset  card*/ 

CI_0pen  /*open  socket*/ 

Cl _Get Configuration  /*check  card's  configuration*/ 

CI_CheckPIN  /*unlock  card*/ 

for  each  certificate  on  the  Card, 

build  a  list:  (personality  label, certificate  index) 

We  found  error  in  this  algorithm,  as  CIJDpen  must  happen  befor-r  ClJteset. 
msp_clear  specification  is  to  “call  the  Cl  functions  to  clear  all  registers,  terminate  the  Fortezza 
library,  and  close  the  Card  socket  ”  [1],  p.25.  This  would  indicate  that  msp_cleax  algorithm  is  as 
follows: 

CI_Reset  /*reset  card*/ 

CI_Terminate  /*terminate  Cl  library*/ 

CI_Close  /*close  socket,  reset*/ 

However,  the  same  page  also  states  that  “msp_clear  logs  the  user  off  the  Card  by  resetting  it, 
thereby  clearing  all  internal  registers.  The  Cl  function  called  by  msp-clear  will  also  close  the  Card 
socket  ”  [1],  p.25.  This  would  indicate  that  msp-cleax  algorithm  is  as  follows: 

CI_Close  (CI_NULL_FLAG)  /*close  socket,  reset*/ 

We  chose  the  last  specification. 


1-11 


4.2  Card,  Cl  Library  and  Socket  States 


Cl  Library  can  be  either  Initialized  (i.e.  ready  for  execution)  or  Uninitialized.  CLLIB-ALRDY  JNIT 
is  the  dag  that  indicates  if  the  library  is  already  initialized  or  not. 

Card  can  be  in  the  following  states: 


Card  State 
CldPowerUp 
Cl_Ze:oize 
CLUiitialized 

CUnitialized 

Cl-SSOInitialized 

CI-LAWInitialized 

CLUserlnitialized 

Cl-Standby 

CLReady 


Description 

self  test,  determine  Card  state 
zeroized 

uninitialized,  or  zeroized  and  PIN 
was  not  entered;  supply  default  SSO  PIN 
init.  params  loaded:  change  SSO  PIN 
SSO  PIN  set;  build  c:  load  certificates 
user  certificates  and  params  loaded: 
install  user  PIN 
supply  PIN  to  log  on 
choose  personality 
execute  security  functions 


Sccket  can  be  in  the  following  states,  assuming  that  socket  is  powered  up  “permanently  (i.e. 
even  if  we  power  it  down,  we  power  it  up  immediatelly): 


Socket  states 

CardNotlnserted 

SocketClosed 

Cl-SccketUnlocked 

CI_HoldLock 

Cl-Sc-cketLocked 


Description 

we  defined  this  state,  to  complete  the  specs 
we  defined  this  state,  to  complete  the  specs 
socket  is  unlocked 

this  application  holds  the  lock  to  the  socket 
socket  is  locked  and  unavailable 


We  defined  the  composite  states  of  the  library  and  the  socket: 

Statel  CI_LIB„ALRDY_INIT  ==  0  &&  socketState [currentSocket]  ==  SocketClosed 


State2  CI_LIB_ALRDY_INIT  !=  0  &&  socketState [currentSocket]  ==  SocketClosed 


State3  CI_LIB_ALRDY_INIT  !=  0  W  socketState [currentSocket]  !=  SocketClosed 

“Curent  socket’’  refers  to  the  socket  currently  selected.  Once  a  socket  is  selected,  either  by 
Cl -Open  or  Cl -Select,  all  subsequent  commands  refer  to  that  socket. 

We  also  have  to  take  into  account  what  actions  a  user  can  execute  based  on  the  physical  condition 
of  the  card.  We  define  states  as  seen  from  the  User’s  point  of  view: 


Use:  states 

in_Statel 

in_S:ate2 

in_State3 

in_S:ate4 


Description 

not  logged  in,  card  not  inserted 
not  logged  in,  card  inserted 
logged  in,  card  not  inserted 
logged  in,  card  inserted 
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These  states  are  valid  for  every  slot  (i.e.  socket)  the  Lser  has  access  to.  Therefore,  we  construct 
a  state  diagram  for  possible  r.ates  of  the  Card  and  sockets  and  user  action  (Fig.  4.1).  This  state 
diagram  is  valid  for  one  socke:.  and  needs  to  be  activated  for  each  socket  separately. 


1:  In_Statel :  no  card  inserted,  not  logged  in 
2:  In_Stau2:  card  inserted,  not  logged  in 
3:  In_State3:  card  not  inserted,  started  to  log  in 
4:  In_State4:  card  inserted,  logged  in 


Figure  4.1:  User  Action  Allowed 


Chapter  5 


SPIN  Specification  of  CI-MSP 
Interface 


This  work  illustrates  a  “divide  and  conquer”  approach  to  specification  and  verification  of  a  large, 
complex  system  like  MISSI. 

We  model  MISSI  system  as  modules,  where  we  abstract  the  modules  which  are  less  important  by 
their  input-output  function,  and  focus  on  the  modules  of  interest.  Therefore,  we  specify  and  verify 
the  entire  system,  by  verifying  the  interaction  between  the  components  as  well  as  the  internal  details 
of  the  components. 

Currently,  we  have  four  different  models  of  MISSI:  [3]  specifies  a  detailed  model  of  MISSI  sender 
with  an  abstraction  of  local  cache  and  certificate  venication;  [4]  also  specifies  the  sender,  but  focuses 
on  the  local  cache  and  certificate  verification,  and  a:  sir  acts  away  the  processes  necessary  to  prepare 
a  message  for  sending:  [5]  abstracts  away  the  selling  process,  and  focuses  on  the  receiver  md 
forwarding.  In  this  model,  we  provide  the  finishing  piece  of  MSP  UA  workstation,  and  specify  -.he 
MSP-CI  function  interface. 

5.1  Card  Swapping  and  Context  Switching 

There  are  several  possible  scenarios  when  working  the  cards: 

•  users  are  free  to  work  with  several  cards  and  :ersonalities  at  a  time. 

•  we  can  have  several  MSP  UAs  working  at  tie  same  time  with  one  card 
reader  (e.g.  several  windows  opened) 

•  we  can  have  several  Cl  libraries  at  the  same  mme. 

In  this  work,  we  assume  one  Cl  library,  one  MS?  UA,  two  sockets,  and  two  cards. 

“Ho;  swap”  denotes  removing  one  card  and  inserung  another,  in  the  middle  of  a  session.  “Coi'.ext 
switching”  denotes  several  MSP  UAs  working  concurrently  with  one  card  reader.  Therefore,  i  we 
use  CCSs'  Composition  operator  to  denote  concurrent  behavior,  our  goal  is  to  specify: 
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MSPJJA1  I  MSP.UA2  I  ...I  HSPJJAn  I 
socketl  I  socket2  I  . . •  i  sccketm  I 
Cl.library 

We  have  specified  hot  swap  npport. 

In  our  model,  we  have  two  sockets: 

MSPJJA1  I  socket 1  I  socket2i  Cl.library 

[10],  p.52,  states:  “Application  should  probably  support  different  startup  processes,  perhaps  one 
for  "cold  start”  (system  boot  and  reboot)  and  another  for  hot  swap  support  (remove  one  card,  insert 

another)” 
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Chapter  6 

Results 


We  tested  our  model  using  a  Linear  Temporal  Logic  «LTL)  formula,  and  an  assert  statement. 

First  we  prove  that,  every  time  mspJLogin  is  executed  correctly,  -.he  user's  card  used  for  krin 
is  in  Standby  state.  We  have  done  this  proof  for  a  card  that  belongs  to  a  non-SSO  user.  SSO  user 
behavior  is  not  specified  completely  in  the  Cl  documentation,  and  needs  some  additional  resear m. 

msp_login !pin_type ,  pin,  socketNo; 
f  rom_msp_login?rsp; 
if 

:  :  (rsp=MSP_0K)  ->  skip;  /*user  will  select  personality*/ 
atoaic{ 

logged_out  =  0;  /*user  logged  in*/ 
called_CI_Reset  =  0; 

} 

if 

:  :  (pin_type_that_passed=USER_PIN_TYPE)  -> 

assert  (  Car dS t at e[CardThat Passed]  =  CI_Standby) ; 

/*mst  verify  this  way  because  Cards  can  be  removed  and  inserted*/ 

: :  (pin_type_that.passed  *=  SS0_PIN_TYPE)  ->  skip; 
fi; 

sync!0;  /* synchronizing  semaphore*/ 
f rom_msp_ua ! MSP_0K ; 

: : (rsp !=MSP_0K)  -> 
f rom_ttsp_ua!MSP_FAIL;  error; 
fi; 

The  synchronizing  semaphore  sync  is  used  to  assure  that  the  test  applies  to  the  desired  card,  it 
is  possible,  after  login,  to  insert  and  remove  any  card  into  the  card  reader  slot.  The  synchronize* 
semaphore  assures  that  we  are  referring  to  the  card  that  was  used  for  the  last  login  process. 

The  next  step  is  to  prove  that,  every  time  a  card  is  in  Standby  mode,  i.e.  a  non-SSO  user  a 
logged  in.  the  user  stays  logged  in  until  card  is  removed  or  reset,  or  user  loggs  out.  We  prove  ue 
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LTL  formula  □  (q  U  R) ,  which  means:  “for  all  states,  it  holds  that  a  q-interval  either  runs  :o  the 
end  of  computation  or  is  terminated  by  an  r-state.’’  We  defined  q  and  r  as  follows  for  socket i: 

q  -  CardState  [Card]  --CI_Stardby  &&  socket  Contains  [0]  ==Card 
r  =  socketContains  [0]  ==  Car dSot Inserted  II  called_CI_Reset  ==  1  |  | 
logged_out  ==  1 

Similarly,  we  define  the  same  tests  for  socketl.  using  socketContainsfl]. 

SPIN  uses  different  algorithms  for  exhaustive  "guaranteed  coverage”  search  or  compressed  “ex¬ 
pected  coverage  search.  Expected  coverage  uses  hashing  techniques  for  state  space  compression 
and  memory  reduction.  The  compilation  option  DBITSTATE  evokes  exhaustive  search.  The  c:mpi- 
lation  option  COLLAPSE  is  one  cf  the  options  that  can  be  used  for  compression.  One  of  the  caveats 
is  that  running  an  LTL  formula  restricts  the  exhaustive  run  to  the  states  matched  by  the  formula. 

Therefore,  we  ran  our  model  in  compressed  search  with  LTL  formula  and  in  exhaustive  search 
without  it.  Exhaustive  search  was  done  by  substituting  DBITSTATE  option  for  COLLAPSE  crtion. 
We  ran  our  model  as: 

spin  -a  main. spin 

gcc  “DMEMCNT=28  -DCOLLAPSE  -DSAFETY  -o  pan  pan. c 

pan  -w23  -m300000 

spin  -t  -p  main,  spin  >  look 

The  results  without  LTL  formula  are: 

Depth=  104301  States=  le+06  Transit ions=  1.26712e+06  Memory^  14.388 
Depth=  120996  States^  2e+06  Trans it ions =  2.74418e+06  Memory^  14.900 
(Spin  Version  3.2.0  —  8  April  1998) 

+  Partial  Order  Reduction 

Bit  statespace  search  for: 
never-claim  -  (none  specified) 

assertion  violations  + 

cycle  checks  -  (disabled  by  -DSAFETY) 

invalid  endstates  + 

State-vector  200  byte,  depth  reached  120996,  errors:  0 
2.37l69e+06  states,  stored 
1.0109e+06  states,  matched 
3 . 38259e+06  transitions  (=  stored+matched) 

9942  atomic  steps 

hash  factor:  3.53698  (best  coverage  if  >100) 

(max  size  2~23  states) 


Stats  on  memory  usage  (in  Megabytes): 
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483.824  equivalent  memory  usage  for  states  (stored* (State-vector  +  overhead)) 
2.097  memory  used  for  hash-array  (-w23) 

8.400  memory  used  for  DFS  stack  (-m300000) 

14.900  total  actual  memory  usage 

details  about  unreached  states  . . . 

The  results  with  LTL  formula  are: 

warning:  never-claim  +  accept-labels  requires  -a  flag  to  fully  verify 
warning:  for  p.o.  reduction  to  be  valid  the  never  claim  must  be  stutter-closed 
(never  claims  generated  from  LTL  formulae  are  stutter-closed) 

Depth=  76218  States=  le+06  Transitions=  1.26914e-Ki6  Memory=  66.559 

Depth=  96626  States=  2e+06  Trans  it  ions  =  2.68848e-06  Memory=  91.033 

Depth=  966c6  States=  3e+06  Transitions=  4.10866e-C6  Memory=  114.995 

Depth=  96626  States=  4e+06  Transitions=  5.4948e+C6  Memory=  139.059 

Depth=  96626  States=  5e+06  Transitions=  6 . 90053e-Ki6  Memory=  163.123 

(Spin  Version  3.2.0  --  8  April  1998) 

+  Partial  Order  Reduction 
+  Compression 

Full  statespece  search  for: 
never-claim  + 

assertion  violations  +  (if  within  scope  of  claim) 
cycle  checks  -  (disabled  by  -DSAFETY) 

invalid  endstates  —  (disabled  by  never-claim) 

State-vector  204  byte,  depth  reached  96636,  errors:  0 
5.65043e+06  states,  stored 
2.17l59e+06  states,  matched 
7 . 82202e+06  transitions  (=  stored+matched) 

40986  atomic  steps 

hash  conflicts:  1.29224e+06  (resolved) 

(max  size  2*23  states) 

Stats  on  memory  usage  (in  Megabytes): 

1220.492  equivalent  memory  usage  for  states  (stored* (State-vector  +  overhead)) 
135.869  actual  memory  usage  for  states  (compression:  11.13%) 

State-vector  as  stored  =  12  byte  +  12  byte  overhead 
33.554  memory  used  for  hash-table  (~w23) 

7.200  memory  used  for  DFS  stack  (-m300000) 

178.687  total  actual  memory  usage 

nr  of  templates:  [  globals  procs  chans  ] 
collapse  counts :  [  1165  242  5064  276  3  43  2  ] 

details  about  unreached  states  . . . 
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Chapter  7 

Cross  Certification  in  X.509 


The  old  lersion  of  SDN  standard,  SDN. 701,  and  X.509  standard  specified  only  certificates  issued  by 
a  single  Root  certification  authority  (PAA). 

The  new  SDN,  SDN. 801,  specifies  certificates  issued  by  several  PA  As.  For  example,  each  of  the 
allied  countries  can  have  a  PAA,  issuing  certificates  fro  that  country.  If  someone  from  the  USA 
wants  to  exchange  secure  mail  with  someone  from,  say,  France,  the  certificates  will  be  issued  by 
different  PAAs. 

Some  of  the  problems  that  we  encounter  in  this  new  approach  include: 

•  In  the  one-PAA  system  the  PAA  was  the  point  of  trust  that  the  entire 
certificate  validation  was  based  on.  In  the  case  of  cross-certificates,  we 
need  to  find  out  how  security  can  be  preserved. 

•  The  access  rules,  such  as  PRBAC,  need  to  be  specified  in  case  of  multiple 
PAAs.  This  task  was  started  for  one-PAA  domains  in  [5].  Additional  work 
needs  to  be  done  for  multiple  PAAs. 

Let  us  assume  that  we  have  users  Userl  and  User2.  Userl  is  certified  by  CA1.  which  is  certified 
by  PCA1.  which  is  certified  by  PAA1.  User2  is  certified  by  CA2,  which  is  certified  by  PCA2,  which 

is  certified  by  PAA2.  Users  1  and  2  can  exchange  secure  mail  iff  PAA1  and  PAA2  are  mutually 
certified. 

Scenario  1:  User2  wants  to  send  mail  to  Userl.  User2  would  then  find  the  certificate  of  Userl 
from  the  public  Directory.  In  fact,  this  certificate  will  be  signed  by  CAA1;  which  means  that  User2 
will  have  to  retrieve  certificate  of  CAA1,  signed  by  PCA1,  and  certificate  of  PCA1  signed  by  PAA1. 

In  the  one-PAA  system,  the  PAA  s  certificate  was  loaded  on  the  users  Care.  The  whole  certi¬ 
fication  procedure  was  based  on  this  certificate  being  securely  and  accurately  stored  on  the  Card. 
Userl  would  have  used  this  certificate  to  unsign  the  certificate  of  PAA1,  then  wodd  use  certificate  of 
PAA1  to  validate  certificate  of  CA1.  and  finally  use  certificate  of  CA1  to  validate  User2’s  certificate. 

In  the  multi- PAA  case,  User2  does  not  have  certificate  of  PAA1  on  his  Card.  Therefore,  the 
validation  process  could  be  completed  iff  User2  can  somehow  obtain  certificate  of  PAA1.  We  propose 
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the  following:  Each  Card  must  b(  loaded  with  the  certificate  of  the  PAA(s)  under  which  the  Card  s 
ouner  resides  ( let  us  call  these  PAAs  “ home  PAAs”),  and  certificates  of  all  other  P.4.4s  cross- certified 
by  the  “home  PAAs.”  For  example,  for  the  above  example,  Card  of  User2  would  have  been  loaded 
with  the  certificate  of  PAA2  and  certificate  of  PAA1  signed  by  PAA2. 
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Chapter  8 

Conclusion 


This  work  illustrates  a  divide  and  conquer  ’  approach  to  specification  and  verification  of  a  large, 
complex  system  like  MISSI. 

We  model  MISSI  system  as  modules,  where  we  abstract  the  modules  which  are  less  important  by 
their  input-output  function,  and  focus  on  the  modules  of  interest.  Therefore,  we  specifv  and  verifv 
the  entire  system,  by  verifying  the  interaction  between  the  components  as  well  as  the  internal  details 
of  the  components. 

Currently,  we  have  four  different  models  of  MISSI:  [3]  specifies  a  detailed  model  of  MISSI  sender 
with  an  abstraction  of  local  cache  and  certificate  verification;  [4]  also  specifies  the  sender,  but  focuses 
on  the  local  cache  and  certificate  verification,  and  abstracts  away  the  processes  necessary  to  prepare 
a  message  for  sending;  [5]  abstracts  away  the  sending  process,  and  focuses  on  the  receiver  and 
forwarding.  In  this  model,  we  provide  the  finishing  piece  of  MSP  UA  workstation,  and  specify  the 
MSP-CI  function  interface. 

What  we  find  is  that  specifications  seem  correct  when  done  in  English,  and  that  the  subtleties 
needed  for  final  implementation  can  be  recoved  only  with  formal  specification.  We  find  that  lack  of 
formal  documentation  leaves  many  unspecified  and  confiding  details.  The  critical  piece  is  always 
the  interface  between  various  modules. 
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The  State-of-the-Art  in  Formal  Methods 


Currently,  there  are  more  than  75  different  formal  methods  (FMsi  listed  on  the  de-facto 
formal  methods  repository  at  the  World  Wide  Web  Virtual  Library  on  Formal  Methods, 
hnp:/A\ww.comlab.ox.ac.uk/an:hive/formal -methods/.  This  is  a  de-facto  database  of  anything 
relating  to  formal  methods,  with  entries  mostly  from  the  USA  and  Europe.  Formal 
methods  are  in  different  stages  of  development,  in  a  wide  spectrum  from  formal 
languages  with  no  tool  support,  to  internationally  standardized  languages  with  tool 
support  and  industrial  users.  The  field  of  formal  methods  is  in  a  great  flux  and  evolving 
rapidly,  leaving  research  laboratories  and  making  inroads  into  industrial  practice. 

One  of  the  major  issues  in  the  technology  transfer  from  academic  research  to  industrial 
practice  is  how  to  make  formal  methods  more  available  to  practitioners.  First,  formal 
methods  are  not  appealing  to  industry  unless  there  are  adequate  tools  that  support  the 
methods.  Second,  since  real  problems  require  diverse  approaches,  and  no  single  formal 
method  is  suitable  for  solving  all  classes  of  problems,  it  seems  necessary  to  take 
advantage  of  different  strengths  of  different  formal  methods.  It  would  be  highly  desirable 
to  integrate  formal  methods,  especially  those  that  have  tool  support. 

We  focus  on  formal  tool  integration  as  a  practical  step  in  a  more  widespread  use  of 
formal  methods.  In  order  to  assess  the  latest  developments  in  the  field  of  formal  methods 
and  tools,  we  participated  in  several  meetings  and  conferences.  Our  goal  was  to 
investigate  possibilities  for  deriving  a  framework  for  integrating  formal  tools  into  a 
toolkit,  either  on  its  own,  or  as  a  part  of  existing  toolkits. 

We  participated  in  the  following  events:  two  conferences  focused  on  formal  methods,  one 
predominantly  software  oriented  (WIFT’98),  and  another  predominantly  hardware 
onented  (FMCAD’98),  a  visit  to  a  commercial  research  facility  (SRI  International);  a 
research  meeting  (DARP  A  PI  meeting);  and  the  IEEE  Computer  Science  Software 
Engineering  Standards  Committee  (SESC)  Formal  Methods  Planning  Group.  This  wide 
variety  of  events  exposed  us  to  academic,  industrial,  and  government  formal  methods 
communities,  including  tool  makers,  tool  users,  and  funding  agencies. 

The  overall  impression  is  that  industry  is  actively  seeking  new  approaches  that  could 
assist  in  producing  complex  products  while  meeting  short  time-to-market  deadlines. 
Industry  is  considering  formal  methods  as  a  systematic  approach  to  dealing  with  the 
overwhelming  amount  of  information.  CAD  industry  seems  the  most  willing  and  already 
widely  uses  many  lightweight  formal  tools  such  as  model  checkers  and  equivalence 
checkers.  Telecommunications  industry  comes  next.  Some  of  the  expressed  needs 
include:  more  user-friendly,  powerful  and  robust  tools;  more  real-life  applications;  more 
infrastructure  such  as  verified  libraries;  more  publicity  of  success  stories  and  available 
technologies;  and  more  training. 
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Proposed  Solutions 

Based  on  our  findings,  we  have  constructed  an  outline  of  steps  that  can  be  taken  to  allow 
for  integration  of  formal  tools  into  existing  industrial  practice. 

Step  1 

The  main  questions  to  be  answered  are: 

•  What  are  the  problems  that  users  need  solved? 

•  What  kind  of  tools  and  features  would  be  useful  to  the  users? 

To  answer  these  questions,  we  must  interview  decision  makers  and  cutting-edge 
researchers  to  find  out  what  direction  the  field  is  moving  in,  and  domain  engineers  to  find 
out  what  the  practicing  engineers  use  and  need  in  their  daily  work.  In  other  words,  we 
need  to  understand  the  current  process  flow  and  directions  in  which  it  is  evolving,  as  well 
as  the  tools  used.  This  task  might  be  difficult  to  realize  in  practice,  because  commercial 
companies  are  very  protective  of  their  process  flow  as  it  contains  trade  secrets.  Off-the- 
shelf  tools  are  highly  customized  to  fit  a  particular  design  flow.  Companies  that  might  be 
more  willing  to  disclose  their  process  flow  are  small  and  medium  size  companies. 

More  specific  questions  that  need  to  be  answered  are: 

•  What  application  domain  should  the  toolkit  address? 

Since  different  application  domains  require  different  notations  and  features,  it  would 
be  the  best  to  develop  a  toolkit  for  one  specific  application.  We  narrow  down  the 
choices  for  relevant  application  domains  to:  telecommunications,  CAD,  and  software 
industries.  Telecommunications  and  CAD  industries  are  the  most  mature  application 
domains  suitable  for  introduction  of  formal  methods  tools.  Software  industry  in 
general  is  not  expressing  sufficient  interest  yet,  except  for  the  safety-critical  software. 

•  What  audience  should  be  targeted  as  users  of  the  formal  toolkits? 

Possible  choices  are:  academic  or  industrial  researchers;  design  engineers;  application 
domain  engineers;  testing  engineers,  or  implementers.  A  related  audience  to  keep  in 
mind  is  tool  developers.  Our  preference  is  that  the  toolkit  should  address  working 
engineers  and  their  daily  needs. 

•  What  level  of  abstraction  should  the  toolkit  address? 

For  example,  should  the  toolkit  address  requirements,  specification,  high-level 
design,  intermediate-level  design,  low-level  design,  testing,  and/or  implementatioa 
Ideally,  the  toolkit  could  be  used  all  the  way  from  top-level  design  to  the 
implementation  and  testing,  in  a  stack-like  fashion,  where  tools  communicate  from 
one  layer  to  another. 

•  What  kinds  of  tools  should  be  integrated  in  the  toolkit? 

The  answer  to  this  question  depends  on  the  answers  to  the  above  questions,  as  well  as 
availability  of  tools  and  possibility  for  collaboration  with  tool  makers.  Possible  choice 
of  tools  is  outlined  in  the  previous  section.  We  feel  that  the  toolkit  must  include 
lightweight  tools,  user-friendly  interface  (preferably  graphical),  and  informal  tools 
that  help  automate  tedious  tasks. 
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A  question  remains  if  the  tools  integrated  should  be  general-purpose  or  specialized.  It 

seems  the  most  effective  to  include  general  purpose  customizable  tools,  so  that  users  can 

tailor  them  to  their  own  needs  and  evolve  the  tools  as  their  design  flow  evolves. 

Based  on  stated  industrial  needs,  we  conclude  that  tools  must  address  the  issue  of: 

•  Clarity,  notation  must  be  accessible  to  engineers  and  have  power  of  expressiveness. 

•  Error  finding:  tools  must  identify  bugs  and  mistakes,  rather  than  to  prove  correctness. 
Errors  must  be  clearly  identified,  i.e.  what  line  of  code,  what  statement,  why  it  is  an 
error. 

•  Effectiveness:  tools  must  be  fast,  easy  to  use,  and  powerful. 

•  Robustness,  the  tool  itself  should  be  as  bug-free  as  possible;  proofs,  analysis,  and 
validation  should  also  be  robust,  i.e.  should  not  easily  break  when  specification 
changes. 

•  Extensibility:  users  must  have  the  ability  to  customize  the  tool  to  a  particular  design 
flow. 

•  Automation  and  methodology:  tools  should  be  as  automated  as  possible.  At  least, 
users  should  have  some  methodology  to  guide  them. 

•  Reusability  and  change:  proofs/modules  should  be  reusable  during  frequent  changes 
in  requirements  and/or  specification. 

•  Scalability:  tools  must  have  some  means  to  organize  large  and/or  growing 
specifications. 

•  Compatibility  with  existing  practices/tools:  compatibility  comes  in  two  forms:  the 

tool  must  be  able  to  “fit”  into  the  existing  design  flow;  and  the  tool  should  resemble 
existing  tools. 

•  Integration  of  graphical  and  textual  notations,  acceptable  user  interfaces. 

•  Proving  application  specific  properties:  tools  must  be  appropriate  for  the  task  at  hand. 

•  Tool  support:  the  tool  must  have  good  technical  support,  user  manual,  documentation, 
upgrades,  and  training. 

•  Stability:  how  likely  is  it  that  the  tool  and  the  tool  vendor  will  exist  in  the  next  five 
years? 


Step  2 

The  main  questions  to  be  answered  are: 

•  What  kind  of  functionality  is  already  existing  in  the  current  formal  tools9 

•  Which  formal  tools  would  satisfy  requirements  outlined  in  Step  1? 

•  How  can  these  tools  be  integrated  into  the  design  flow  outlined  in  Step  1? 

Understanding  formal  tools  requires  creating  taxonomy  and  classification  of  the  current 
formal  tools,  clearly  outlining  capabilities,  inputs  and  outputs.  This  work  would  take 
some  time  and  effort,  and  is  recognized  by  the  SESC  Formal  Methods  Planning  Group  as 
a  very  much  needed  step  in  identifying  what  formal  tools  we  have  available  to  work  with. 
Tool  makers  are  requesting  this  information  in  order  to  be  able  to  integrate  the  tools.  It 
would  be  most  effective  to  start  this  work  with  a  selected  subset  of  tools. 
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We  also  need  to  evaluate  a  selected  subset  of  tools  to  identify  those  suitable  for  inclusion 
in  the  toolkit.  We  must  examine  if  a  tool  is  robust  enough  for  industrial  use.  This  criteria 
includes: 

•  Efficient  and  bug-free  execution:  the  tool  must  be  sufficiently  tested  by  a  variety  of 
users,  reported  bugs  must  be  fixed  quickly,  efficiency  needs  to  be  improved,  and  the 
improved  tool  distributed.  It  is  usefUl  to  have  tools  and  updates  downloadable  from 
the  web,  and  to  have  user  mailing  lists  and  user  meetings. 

•  Documentation:  usually  one  of  the  weakest  links.  User  manuals  are  frequently  not 
clear  and  complete,  because  they  are  written  by  tool  makers  who  understand  the  tool 
so  well  that  they  take  the  information  for  granted,  or  do  not  have  the  time  or  the 
preference  to  write  reports.  Error  messages  generated  by  the  tools  also  need  to  be 
improved,  to  show  where  and  what  occurred.  This  work  requires  many  days  of 
“grange”  work  that  is  not  publishable  and  not  well  represented  in  grant  funding.  A 
commercial  development  team  could  implement  these  user-friendly  features. 

•  Technical  support:  it  is  necessary  to  have  dedicated  personnel,  well  versed  in  the  tool, 
available  to  answer  user  questions.  Clearly,  this  takes  personnel  time. 

•  Learning  curve  and  user  friendliness:  the  tool  has  to  use  notations  that  are  easy  to 
learn  by  the  users,  because  industrial  users  do  not  have  the  time  to  learn  complex 
tools  on  the  job.  Until  more  users  are  trained  in  formal  tools,  we  need  to  resort  to  this 
measure. 

Other  critical  issues  for  success  of  the  toolkit  include: 

•  technology  transfer 

•  publicity  and  marketing 

•  extensibility  of  the  tool,  i.e.  users  must  be  able  to  customize  the  tool. 

Understanding  how  the  tools  can  be  integrated  might  be  difficult.  The  major  barrier  is 
that  it  is  impossible  to  obtain  source  code  for  many  tools.  Even  if  source  code  is  obtained, 
often  collaboration  with  the  tool  maker(s)  is  needed.  In  addition,  it  is  common  that  each 
company  highly  customizes  off-the-shelf  tools  to  fit  into  the  company’s  design  flow.  If 
we  cannot  obtain  tools  that  we  could  build  on,  we  could  examine  source  code  of  CAD 
tools  constructed  for  government  projects. 


Conclusion 

Industry  needs  assistance  in  producing  complex,  bug-free  products  while  meeting  short 
time-to-maiket  deadlines.  Industry  is  considering,  or  already  using,  formal  methods.  For 
example,  many  large  companies  have  internal  formal  methods  groups,  and  CAD/CASE 
toolkits  include  model  checkers  and  equivalence  checkers.  Formal  methods  are  widely 
accepted  in  hardware  engineering,  and  less  so  in  software  engineering.  In  any  case, 
formal  methods  have  not  reached  a  maturity  stage  yet  and  are  in  transition  from  research 
to  practice.  The  consensus  is  that  this  transition  can  be  aided  by  producing  industrial 


1-27 


strength  formal  methods  tools,  more  technical  infrastructure  such  as  verified  libraries, 
more  (publicized)  industrial  case  studies,  and  increasing  the  pool  of  trained  users.  It  is 
often  stated  that  formal  methods  and  tools  must  be  accessible  to  engineers  through  their 
notation,  features,  and  approach. 

Therefore,  we  must  come  up  with  a  strategy  that  will  effectively  use  formal  tools  within 
the  currently  existing  parameters,  and  work  towards  longer-term  goals.  We  feel  that,  at 
this  time,  it  seems  most  practical  to  have  lightweight  formal  tools  working  in  conjuction 
with  the  existing  tools  and  practices.  This  approach  will  demonstrate  usefulness  of  formal 
methods  in  addressing  problems  not  easily  addressable  otherwise,  and  allow  the  users  to 
get  accustomed  to  formal  methods’  paradigms.  More  heavyweight  tools  would  be  used 
by  a  small  group  of  experts.  Meanwhile,  a  feedback  cycle  could  be  started:  the  need  for 
better  tools  and  positive  experiences  with  the  existing  tools  will  cause  greater  industrial 
demand,  so  that  academia  will  be  willing  to  train  more  users;  and  vice  versa.  Unless  there 
are  more  trained  users  and/or  need,  widespread  use  of  formal  methods  will  always  stay  at 
the  lightweight  level. 

We  can  recommend  several  measures  that  could  improve  the  current  situation  and 
increase  industrial  use  of  formal  methods: 

•  Provide  funding  for  tools  thai  are  transitioning  from  research  environment  into 
production  environment.  This  would  be  an  appropriate  role  for  the  government. 

•  Create  opportunities  for  user  training,  either  by  influencing  the  academia  or  providing 
independent  user  training.  An  appropriate  role  for  the  government  would  be  to 
provide  training  opportunities  and  thus  break  the  chicken-and-the-egg  cycle  of 
academia  waiting  for  a  larger  industrial  demand,  and  industry  demanding  more  when 
there  are  more  trained  users  demanding  more  tools. 

•  Generate  publicity  for  formal  tools  and  research,  so  that  the  available  knowledge  can 
be  disseminated  through  wider  circles  of  users  and  different  communities.  For 
example,  cross-pollinating  research  in  hardware  and  software  protocols  would  be 
useful  to  both  communities.  We  recommend: 

•  one-page  fliers  to  be  distributed  at  conferences 

•  tool  demos  at  a  wide  array  of  both  academic  and  commercial  conferences 

•  a  web  page,  linked  to  the  World  Wide  Web  Virtual  Library  on  Formal  Methods, 
with  clear  taxonomy  of  terms  used  in  formal  methods  and  classification  of  the 
tools 

•  a  web  page,  linked  to  the  World  Wide  Web  Virtual  Library  on  Formal  Methods 
page,  with  documentation  of  case  studies,  especially  success  stories.  Publicized 
success  stories  could  be  used  to  justify  use  of  formal  methods. 

•  a  conference  to  present  SRI  International  and  other  research  of  interest  to  the 
Rome  Research  Site  to  a  wide  community.  The  purpose  of  this  conference  would 
be  to  disseminate  information  and  to  provide  a  forum  for  discussion  and  exchange 
of  ideas. 

•  Develop  taxonomy  of  terms  used  in  formal  methods  and  classification  of  formal 
tools,  clearly  stating  tools’  capabilities.  Ideally,  we  would  include  tools’  inputs  and 
outputs.  This  survey  needs  to  be  well  publicized,  including:  a  web  page  linked  to  the 
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World  Wide  Web  Virtual  Library  on  Formal  Methods  page;  notices  to  e-mail  lists; 
notices  to  newsgroups;  and  one-page  fliers  to  be  distributed  at  conferences.  This 
survey  would  greatly  benefit  anyone  who  is  searching  for  the  right  tool  as  well  as 
toolkit  makers.  This  effort  would  take  a  log  time  to  complete  fully,  and,  therefore, 
could  be  started  for  a  subset  of  tools  and/or  subset  of  properties. 

•  Create  verified  libraries  and  other  supporting  theories,  in  order  to  build  infrastructure 
to  make  formal  tools  easier  to  use.  Additional  research  would  determine  the  language 
and  contents  of  the  libraries. 

•  Integrate  formal  tools  into  existing  tools  that  are  already  used  in  industry.  This 
approach  would  require  either  access  to  tools’  source  code,  or  extensible  tools. 
Industry  might  not  be  willing  to  disclose  their  design  flow,  and  additional  research 
needs  to  be  done  to  determine  what  kinds  of  tools  are  needed. 

•  Create  a  framework  to  integrate  formal  tools  into  usable,  industrial  strength  toolkits. 
Some  work  is  already  being  done  in  this  arena,  mostly  in  Europe  (please  refer  to  the 
list  of  related  projects).  Integrating  different  tools  is  a  difficult  undertaking  that  might 
possibly  requires  access  to  the  source  code  of  various  tools  and  collaboration  with  the 
original  tool  developers.  The  feasibility  of  the  framework  would  be  best  illustrated  by 
including  parts  of  a  tool  into  an  existing  and  available  toolkit. 


Overview  of  Some  Commonly  Used  Formal  Tools 

Toolkits  Which  Include  Model  Checkers 

Model  checkers  are  automated  tools  based  on  state  space  exploration.  There  are  many 
model  checkers  in  current  use.  They  have  become  quite  popular  in  the  recent  years,  and 
are  claimed  to  have  “saved  the  reputation  of  formal  methods”  because  of  their  ease  of  use 
and  learning.  Model  checkers  are  automated  tools  suitable  for  users  not  trained  in 
rigorous  mathematics  and  theorem  proving  skills.  Currently,  model  checkers  are  widely 
used  in  industry  and  in  academia.  Some  of  the  most  popular  ones  in  the  USA  include,  in 
alphabetical  order:  Murptp,  SCR,  SDT,  SMV,  and  SPIN.  Model  checkers  popular  in 
Europe  are  FDR,  LOTOS,  SDT,  and  VDM  tools. 

The  variety  in  model  checking  tools  is  due  to  the  fact  that  tool  makers  had  different  use 
of  the  tools  in  mind.  Roughly  speaking,  we  can  categorize  model  checkers  based  on  the 
language  supported  (which  is  based  on  application  domain,  level  of  use,  and  intended 
audience),  and  measures  used  to  achieve  efficiency.  Priorities  for  choosing  or  creating  a 
language  could  include: 

•  Simple  notation,  such  as:  programming-like,  tabular,  or  graphical 

•  Concurrency 

•  Message  communication 

•  Message  buffering 

•  Value-passing  (i.e.  use  of  data) 
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•  Equivalence  between  various  specifications 

•  Synchronous  versus  asynchronous  modeling 

The  language  is  also  catered  to  the  specification  level  it  will  be  applied  to  (e.g. 
requirement  specification).  Most  model  checkers  are  intended  for  describing  control¬ 
intensive  systems  and  usage  by  “regular”  engineers.  Most  model  checkers  include  a 
symbolic  simulator  (it  is  a  byproduct  of  storing  state  space),  and  temporal  logic  for 
describing  system  properties. 

Measures  for  increasing  efficiency  usually  involve  the  language  in  which  the  tool  is 
implemented  (such  as  C,  Lisp,  or  ML)  and  algorithms  for  reducing  state  space,  such  as 
exploiting  symmetry  (Murptp),  hash  reduction  (SPIN),  or  automatic  abstraction  of 
unnecessary  details  (SCR*). 

Murptp 

http://verifv7stanford.edu/dill/murphi.html 

Murptp  toolset  is  intended  for  asynchronous,  interleaving  modeling  of  concurrent 
systems,  particularly  hardware  protocols.  Murptp  toolset  is  based  on  Murptp  language, 
which  is  similar  to  Lisp  programming  language.  Murptp  language  has  constant,  type, 
variable  and  procedure  declarations,  rule  definitions,  a  description  of  the  next  state,  and  a 
collection  of  invariants  (Boolean  expressions  that  reference  a  variable.)  Murptp  includes  a 
model  checker.  It  is  possible  to  check  for  invariant  violations,  error  statements,  assertion 
violations,  and  deadlock.  It  is  possible  to  interface  to  SVC.  Methods  for  increasing 
efficiency  of  Murptp  include:  state  reduction  by  using  symmetry,  execution  of  action 
rules  in  reverse  order,  and  keeping  track  of  the  number  of  processes  in  a  particular  state 
instead  of  the  processes  themselves. 

Symbolic  Model  Verifier  (SMV) 

ftp://emc.cs.cnm.edu/pub/ 

SMV  was  intended  for  hardware  systems.  The  SMV  model  checker  is  based  on  language 
SMV,  which  hierarchically  describes  finite-state  machines  at  any  level  of  detail,  both 
synchronous  and  asynchronous.  SMV  uses  CTL  for  verification.  CTL  is  an  extension  of 
Boolean  logic  with  four  temporal  formulas:  atomic  formulas,  propositional  logic,  next- 
state  logic,  and  “until”  logic.  CMU  SMV  is  implemented  using  BDDs.  There  is  also 
SMV  tool  from  CadenceBerkeley  Labs. 

Software  Cost  Reduction  (SCR*)  toolkit 

http://www.csr  ncl.ac.uk/proiects/FME/InfRes/tools/frntdb044.  html, 
http://vvAvw.itd.nrl.navv.mil/nD/5340/personnel/heitmever.htinl. 

SCR*  is  intended  for  modeling  safety-critical  software  systems  at  the  level  of 
requirements  specification.  It  is  meant  to  be  used  by  application  engineers  with  no 
training  in  formal  methods  and  no  access  to  formal  methods  experts.  SCR*  uses  SCR 
tabular  notation  to  describe  required  system  behavior  as  the  synchronous  composition  of 
non-deterministic  environment  and  deterministic  system.  Each  SCR  specification  is 
organized  into  two  parts:  dictionaries  (which  define  static  information  such  as  variable 
names),  and  tables  (which  specify  how  the  variables  change  in  response  to  the  input 
events.)  the  SCR*  toolkit  contains  the  model  checker  SPIN  and  several  lightweight  tools: 
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•  Specification  editor 

•  Graphical  dependency  browser,  aids  I  understanding  dependencies  among  variables 

•  Consistency  checker:  exposes  syntax  errors,  variable  name  discrepancies,  missing 
cases,  unwanted  non-determinism,  and  circular  definition. 

•  Simulator 

It  is  possible  to  automatically  generate  state  invariants  (i.e.  properties  that  hold  true  in  a 
given  state)  from  the  requirements  specification.  Before  calling  SPIN,  it  is  possible  to 
automatically  reduce  state  space  by  eliminating  unnecessary  details. 

SPIN 

http:/A\A\^v. dcs.gla.ac.uk/~tfrn/fmt/hol.hiiTil 

SPIN  is  Bell  Labs’  tool  intended  for  concurrent  systems,  particularly  communication 
protocols.  The  underlying  language  is  PROMELA  (Protocol  or  Process  Meta  Language), 
which  is  a  C-like  non-deterministic  language  based  on  Hoare’s  CSP.  SPIN  contains  a 
symbolic  simulator,  model  checker,  and  verification  with  linear  temporal  logic  and 
invariants.  SPIN  uses  hash  algorithms  to  reduce  state  space.  SPIN  is  implemented  in  C, 
and  its  source  code  is  publicly  available. 

SDT  toolkit  is  described  in  Appendix  B  as  a  part  of  Telelogic  Tau  toolkit. 

CAD/CASE  formal  tools  will  be  described  in  Appendix  B. 

Theorem  Provers 

What  we  call  “theorem  provers”  are  actually  mechanized  proof  assistants.  These 
automated  tools  are  powerful  in  their  ability  to  specify  and  verify  large  amounts  of  data, 
and  require  at  least  six  months  of  learning  time. 

A  Computational  Logic  for  Applicative  Common  Lisp  (ACL2) 

http://\viTO.cs.u\vTo.edu/— co\vles/acI2 

ACL2  uses  first-order  logic  of  total  recursive  functions.  The  syntax  is  that  of  Common 
Lisp,  which  allows  interface  to  programming  in  Lisp.  The  ACL2  theorem  prover  can  be 
run  in  completely  automatic  mode  or  with  user  interaction. 

Prototype  Verification  System  (PVS) 

http://pvs.csl.sri.com/ 

PVS  uses  a  language  based  on  a  variant  of  higher-order  logic.  It  contains  a  theorem 
prover,  interface  to  temporal  model  checking,  and  consistency  and  completeness  check 
tools  for  tabular  specifications  of  the  kind  advocated  by  Pamas. 

HOL 

http://www.  dcs.gla  ac.uk/-tfm/fmt/hol. ht^ 

The  HOL  theorem  proving  system  is  based  on  higher  order  logic,  and  uses  Standard  ML 
of  New  Jersey  (SML/NJ)  as  the  specification  language.  Since  SML/NJ  is  public,  HOL  is 
extensible. 
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Stanford  Validity  Checker  (SVC) 

htip://agamemnon. Stanford  edu/-le\ittArc/indev±£mI 

SVC  is  intended  to  be  used  as  a  fast  decision  procedure  for  validating  logical  expressions, 
and  was  used  successfully  in  that  capacity  in  PVS.  A  PVS  proof  would  be  reduced  to  a 
smaller  subgoal,  and  SVC  can  be  used  on  that  subgoal.  SVC  can  be  used  for  static 
analysis  of  formal,  high-level  specifications  of  safety  critical  software.  SVC  is  based  on 
first-order  logic.  The  Web  page  has  little  information  about  the  other  features  of  the  tool. 

Consistency  Tools 

Most  toolkits  that  include  model  checkers  either  include  consistency  tools  or  can  be  used 
for  consistency  checking.  Theorem  provers  also  can  be  used  for  consistency  checking. 
Therefore,  we  could  use  SMV,  PVS,  SCR,  HOL,  and  many  other  tools. 


Formal  Verification  Tools  in  Hardware  Toolkits 

There  are  many  formal  hardware  tools,  but  we  will  focus  on  several. 

Design  Suite  of  Tools,  by  Chrysalis  Symbolic  Design,  Inc.,  htmV/www.chwsalis.com. 
Chrysalis  specializes  in  formal  tools  for  CAD,  and  its  entire  product  line  consists  of  three 
formal  tools.  The  tools  can  operate  independently  or  as  a  toolkit,  either  stand-alone  or  as 
a  part  of  a  CAD/CASE  toolkit.  The  interoperability  with  regular  CAD/CASE  tools  is 
achieved  by  using  Verilog,  VHDL  and  ED  IF  for  input/output  to  Design  tools. 

Design  VERIFYer  is  a  formal  equivalence  checker.  Chrysalis  markets  it  as  a 
replacement  for  gate-level  regression  simulation.  The  tool  can  perform  hierarchical 
verification:  RTL  to  RTL,  RTL  to  gate,  gate  to  gate  or  RTL  to  switch.  The  block  diagram 
of  the  tool  is  given  below. 


input  into  Design  EXPLORE 


*  convert  into  Chrysalis  logic  representation 
**  compare  the  logic  driving  each  state  bit 

Design  EXPLORE  is  an  interactive  debugger,  which  takes  as  input  RTL,  netlist,  or 
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library  sources  in  VHDL,  Verilog,  EDIF,  or  HDL  and  converts  them  into  Chrysalis  logic. 

Design  INSIGHT  Assertions  is  a  model  checker  which  uses  assertions  to  prove  desired 
properties.  Assertions  are  Boolean  statements  that  are  true  at  certain  points  in  code. 
Assertions  can  be  used  to  state  properties  to  be  proved,  such  as  input  and  output  module 
requirements,  and  to  state  assumed  properties  that  hold  true.  Properties  can  be  expressed 
in  VHDL,  Verilog,  or  tool  command  language  (tel).  Temporal  operators  are  also 
available.  Chrysalis  is  marketing  this  model  checker  as  a  time-saving  tool,  because 
modules  can  be  tested  before  the  whole  design  is  operational.  Most  test  benches  operate 
at  the  chip  level,  so  all  modules  have  to  working  in  order  to  test  the  logic. 

The  following  is  taken  from  Chrysalis’  press  release: 

November  16, 1998  -  Toshiba  America  Electronics  Corporation  (TAEC)  has 
selected  Chrysalis’  Design  VERIFYer  equivalency  checker  for  their  gate-to-gate 
formal  verification.  “We  were  spending  thousands  of  hours  an  gate-level 
simulation  for  regression  analysis  and  needed  to  streamline  our  design  process 
through  the  use  of  formal  verification.  Chrysalis'  Design  VERIFYer  is  a  proven 
solution  for  gate-to-gate  comparisons,”  said  Jeff  Berkman,  vice  president,  SLI 
engineering  for  TAEC.  “Additionally,  Chrysalis'  application  expertise  and 
support  is  excellent.” 


FormalCheck,  by  Lucent  Technologies  Bell  Labs  Design  Automation, 
httt>://www. lucent.com/.  FormalCheck  is  based  on  VHDL/Verilog.  The  following  is  an 
excerpt  from  the  hardware  industry  news: 

“Lucent  Technologies  Inc.'s  Bell  Labs  Design  Automatical  (Murray  Hill,  N.J)  has 
been  awarded  a  contract  for  more  than  $1 .25  million  by  Silicon  Graphics/Cray 
Research  (Mountain  View,  Calif.)  for  the  follow-up  purchase  of  additional 
licenses  of  FormalCheck,  its  system-level  model-checking  technology.” 

The  tool  has  been  given  to  Cadence  Berkeley  Laboratories,  and  we  could  not  find 
current  tool  brochures. 

Static  Verifyer,  by  Synopsys  Inc.,  hnp:/A^^'.sNTior^s.com/pro<±:cts/static\erii/stat] c%erif.html. 
Synopsys  is  marketing  Static  Verifyer  toolkit  approach  as  the  substitute  for  the  current 
practice.  Currently,  engineers  do  not  simulate  much  at  the  RT  level,  but  simulate 
extensively  at  the  gate-level.  Full  chip  gate-level  simulations  run  for  days,  up  to  several 
weeks.  Synopsys’  solution  is  to:  simulate  at  the  RT  level  using  Synopsys’  VCS  (Verilog 
simulator)  and  Cycloae  (VHDL  simulator),  and  perform  static  verification  at  the  gate 
level  using  Static  Verifyer.  Static  verification  consists  of  static  timing  and  equivalence 
checking.  Prime  Time  is  Synopsys’  static  timing  tool,  which  checks  if  the  delays  of  all 
paths  in  a  circuit  satisfy  timing  assertions.  Synopsys  Formality  is  equivalence  checker, 
which  compares  functional  equivalency  of  the  RTL  source  to  the  post-synhesis  netlist. 
The  last  part  of  the  Static  Verifyer  toolkit  are  tools  used  for  testing:  Design  Compiler 
Expert  Plus,  Test  Gen.  and  PathTest. 

Static  Verifyer  requires  synthesis-based  design  flow  and  synchronous  design. 
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Eagle™,  by  Synopsys  Inc.,  htto:/Awwv.svnopws.com/products/lnvsvv/eagle  ds.html .  Eagle™  is  a 
toolkit  for  hardware-software  co-design,  intended  for  be  used  as  a  part  of  XRAY® 
Debugger  software  development  toolkit  from  Mentor  Graphic  Corp..  Debugger  works 
with  Microtec  C  and  C++  compilers.  Eagle™  toolkit  consists  of  VHDL  and  Verilog 
simulators  and  VERA™ 


Summary  of  Related  Projects 

Telelogic  Tau  toolkit,  by  Telelogic  Inc.,  http:/Avww. telelogic.com.  This  is  a  commercial 
toolkit  for  all  phases  of  software  development.  Telelogic  Inc.  is  a  Sweden-based  company 
with  offices  in  New  Jersey,  California  and  Illinois.  The  entire  toolkit  costs  about  510,000. 
The  tools  are:  ORCA  for  UML-based  requirement  specification,  SDT  for  system  design 
and  implementation  in  SDL,  and  ITEX  fortesting  with  TTCN.  SDL  is  a  graphical 
internationally  standardized  formal  language  by  ITU.  SDL  is  an  object  oriented  language 
that  can  describe  concurrent  finite  state  machines.  It  accepts  data  inputs  as  abstract  data 
type  (ADT)  or  Abstract  Syntax  Notation  One  (ASN.  1).  The  Telelogic  Tau  toolkit 
automatically  generates  executable  code,  and  automatically  generates  executable  test 
vectors.  SDL  and  TTCN  have  the  ability  to  work  with  ASN.  1 .  SDT  supports  writing 
specifications  of  CORBA  objects  in  Interface  Description  Language  (DDL).  An 
observation:  several  months  after  viewing  the  Telelogic  Tau  demo  at  WIFT’98,  we  were 
contacted  by  a  sales  person.  We  were  impressed  by  Telelogic’s  commitment  to  sales  and 
marketing. 

A  similar  toolkit,  called  ObjectGEODE,  based  on  OMT,  SDL,  and  MSC,  is  pro&iced  by 
Verilog,  USA/France. 

Open  Mechanized  Reasoning  Systems  (OMRS)  project,  by  John  McCarthy,  at  all. 
htto:.- Awwv.mTg.dist.unige.it/omrs/index.html.  This  Internet  site  cannot  be  opened,  and 
information  would  have  to  be  obtained  in  another  way. 

The  15th  International  Conference  on  Automated  Deduction  (CADE- 15),  July  5-10, 
1998,  Lindau,  Germany.  Topic:  Integration  of  Deduction  Systems. 
http:,  /i  1 2 mwv.  ira.uka.de/W  orkshop/cfo.  html. 

Proof  and  Specification  Assisted  Design  Environments  (PROSPER)  Project, 

http:  Awwv.dcs.gla.ac.uk/prosper.  This  is  a  recently  started  3-year  joint  project  between  the 
Universities  of  Glasgow,  Cambridge,  Edinburgh,  and  Karlsruhe,  and  commercial  tool 
builders  Prover  Technology  AB  (Sweden)  and  IFAD  (Denmark).  Prover  Technology  AB 
sells  theorem  proving  technology,  training  and  research.  Prover  Technology  AB  invented 
StHmarck’s  algorithm,  which  became  very  popular  in  Europe  because  it  provides  a  fast 
and  efficient  alternative  to  BDDs  for  verification  purposes.  Prover  Technology  AB 
implements  this  algorithm  in  its  Otter  theorem  prover,  and  was  able  to  verify  formulas  up 
to  350,000  connectives  [FMCAD’98],  p.82.  IFAD  produces  the  VDM-SL  Toolbox, 
currently  used  at  more  than  50  sites,  mostly  in  large  European  companies.  Both 
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companies  plan  to  use  the  PROSPER  toolkit  to  improve  their  commercial  products.  The 
goal  of  PROSPER  project  is  to  have  a  user-friendly,  flexible,  extensible  toolkit  that  can 
easily  integrate  with  various  commercial  tools.  The  PROSPER  toolkit  will  provide  a  GUI 
interface  and  a  common  theorem  proving  support  for  a  software  and  a  hardware  toolkit. 
The  software  toolkit  is  the  VDM-SL  toolbox  produced  by  IFAD.  The  hardware  CAD 
toolkit  will  be  built  by  PROSPER  and  will  be  based  on  VHDL/Verilog.  Other  features  of 
the  project  include:  next  generation  interfaces  and  tools  for  requirements  specification, 
using  natural  language  and  timing  diagrams;  and  open  proof  architecture,  containing  an 
API  for  CAD/CASE  tool  integration,  the  core  proof  engine,  and  a  plug-in  interface. 

Universal  Formal  Methods  Workbench  (UniForM),  http:/Avw-ft  .informatik.uni- 
bremen.de/~agbkb/UniForM .  This  is  a  (completed?)  3-year  joint  project  between  the 
University  of  Bremen  and  Carnegie  Mellon  and  Stanford  Universities,  and  companies 
SRI  International  and  the  Kestrel  Institute.  The  goal  is  to  perform  application-driven 
basic  research  and  develop  reliable  software  for  industry.  The  project  will  result  in  a  tool 
kit  with  a  common  user  interface,  to  be  available  in  public  domain  in  1998.  “UniForM  is 
a  specific  framework  instantiated  with  specific  tools  to  handle  communicating  distributed 
systems  and  real  time  requirements.”  The  project  includes  a  case  study,  control  of  a 
safety  critical  system,  and  using  the  example  of  developing  a  decentralized  central  unit 
for  a  single  track  tramway  networks.  This  project  follows  the  V-model,  developed  by  the 
German  Ministry  of  Interior  and  accepted  as  standard  in  German  software  industry.  V- 
model  is  a  standard  development  process  model  for  planning  and  executing  information 
technology  projects.  UniForM  combines  duration  calculi,  CSP  (with  FDR),  Z  (with  type 
checker)  and  PLC.  The  tools  are  integrated  using  development  by  transformation  from 
one  tool  to  another. 

ESPRESS  project,  http:  'wTvw.first.gmd.de/~cspress.  This  project  was  sponsored  by  Daimler- 
Benz  AG,  and  carried  through  cooperation  between  German  government  and  universities. 
ESPRESS  is  designed  for  developing  software  for  complex,  safety-critical  embedded 
systems,  such  as  intelligent  cruise  control  systems.  The  toolkit  covers  the  entire  product 
cycle  in  application  areas  of  automobile  electronics  and  traffic  light  control.  ESPRESS 
toolkit  provides  one  interface  to  many  tools:  Statecharts  and  Z  notation  are  used  to 
represent  requirements,  Isabelle/HOL-Z  tool  is  used  for  validation,  verification  and 
generation  of  test  cases,  and  commercial  tools  such  as  Statemate  are  used  for  editing,  type 
checking,  and  specification  validation.  ESSPRESS  was  presented  at  the  Z  User  meeting 
(ZUM),  International  Conference  ofZ  Users,  September  1998,  Berlin,  Germany.  The 
conference  was  sponsored  by  Daimler-Benz. 

Project  KorSys  (Korrekte  Software  fur  Sicherheitskritische  Systeme,  or  Correct 
Software  for  Safety-critical  Systems),  http://www4 . informatik.tu- 
muenchen.de/proi/korsWallmdex.html.  is  a  joint  effort  between  companies  BMW,  Siemens 
AG,  FZI,  and  ESG,  and  universities  Technical  University  of  Munich,  and  the  University 
of  Oldenburg.  This  project  deals  with  methods  and  tools  for  reactive  and  finite-state 
systems.  Application  areas  include  avionics  system  at  ESG  and  locking  and  hybrid 
systems  at  BMW.  The  project  page  contains  little  information  in  English. 
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Ptolemy  Project,  http://otolemv.eecs.berkelev.eda7.  This  is  University  of  California  at 
Berkeley  ongoing  effort,  led  by  Edward  Lee.  The  project  includes  a  model  checker 
MOCHA  by  Tom  Henzinger  from  UC  Berkeley,  but  is  mainly  integrating  non-formal 
tools.  Nevertheless,  the  project  is  of  interest  to  us  because  it  provides  expertise  in 
integration  of  diverse  methods  and  tools.  The  project’s  goal  is  to  develop  techniques  ad 
tools  for  modeling  heterogeneous,  reactive  systems,  particularly  embedded  systems. 
Ptolemy  can  model  heterogeneous  systems  such  as  those  including  hardware  and 
software,  analog  and  digital,  and  electrical  and  mechanical  devices.  Ptolemy  can  also 
model  systems  that  are  complex  in  the  sense  that  they  mix  widely  different  operations, 
such  as  signal  processing,  feedback  control,  sequential  decision  making,  and  user 
interfaces.  The  project’s  web  page  claims  that  “using  Ptolemy  software,  a  high-level 
dataflow  model  of  a  signal  processing  system  can  be  connected  to  a  hardware  simulator 
that  in  turn  may  be  connected  to  a  discrete-event  model  of  a  communication  network.  It 
would  be  worth  further  research  to  investigate  how  the  Ptolemy  Project  could  provide  us 
with  a  wealth  of  experience  in  tool  integration.  Ptolemy  research  influenced  products  by 
Cadence,  Motorola,  HP,  and  many  other  companies. 


Formal  Methods  Europe  (FME),  http:/AvwAv.cs.tcdie/FME/,  is  a  European  organization 
supported  by  the  Commission  of  the  European  Union,  with  the  mission  of  promoting  and 
supporting  the  industrial  use  of  formal  methods  for  computer  systems  development. 

Forschungszentrum  Informatik  (the  Research  Center  for  Information  Technologies,  FZI), 
at  the  University  of  Karlsruhe,  Germany,  http: //wwav  .  f/ 1  de/prost/prost  e.html.  FZI  is  an 
industry-oriented  research  institution  with  the  mission  to  implement  technology  transfer 
from  the  university  to  the  industry.  It  is  not  entirely  clear  from  the  web  page,  but  it 
appears  that  FZI  is  a  commercial  company  rather  than  government  or  university 
sponsored  center.  The  FZI  Software  Engineering  Department  focuses  on:  the 
development  of  methods  and  tools  for  the  economical  development  and  use  of  software; 
the  methodology  of  design  and  constitution  of  reusable  software;  development  of  open 
programming  environments;  integration  strategies  of  software  tools;  practical 
applicability  of  formal  methods;  and  the  reorganization  and  re-engineering  of  legacy 
systems. 
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Abstract 


We  consider  the  problem  of  adaptive  detection  of  a  Spread-Spectrum  (SS)  signal  in  the 
presence  of  unknown  correlated  SS  interference,  multipath  signal  reception,  and  additive  im¬ 
pulsive  noise.  The  proposed  general  SS  receiver  structure  is  comprised  by  a  vector  of  adaptive 
chip-based  non-linearities  followed  by  an  adaptive  Auxiliary- Vector  linear  tap-weight  filter. 
The  non-linear  receiver  front-end  adapts  itself  to  the  unknown  prevailing  noise  environment 
providing  robust  performance  over  a  wide  range  of  underlying  noise  distributions  while  the 
adaptive  Auxiliary- Vector  linear  tap-weight  filter  allows  rapid  adaptation  for  SS  interference 
suppression  with  a  limited  data  record.  We  examine  a  clipper-type,  a  puncher-type  and  a 
Hampel-type  non-linearity.  Numerical  and  simulation  results  demonstrate  the  performance 
of  the  proposed  approach  and  offer  comparisons  with  the  conventional  Minimum- Variance- 
Distortionless-Response  (MVDR)  filter,  as  well  as  the  MVDR  filter  preceded  by  a  vector  of 
adaptive  chip-based  non-linearities.  The  latter  structure  is  a  generalization  for  the  multipath 
case  of  the  scheme  originally  proposed  in  [8],  [9]. 
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ROBUST  SPREAD  SPECTRUM  COMMUNICATIONS: 


ADAPTIVE  INTERFERENCE  MITIGATION  TECHNIQUES 


Stella  N.  Batalama 


I.  Introduction 

Signal  detection  in  the  presence  of  impulsive  channel  noise  has  been  considered  in  [l]-[5],  while 
direct-sequence  spread-spectrum  (DS-SS)  signal  detection  under  similar  channel  models  has 
been  studied  in  [6]- [9] .  Receiver  proposals  in  [6], [7]  involve  the  use  of  either  a  conventional 
matched  filter  or  a  majority-vote  receiver  (hard-limiter  non-linearity  per  chip  followed  by  sig¬ 
nature  matched-filtering).  In  [6]  it  is  reported  that  neither  one  of  the  above  proposals  is  uni¬ 
versally  effective  against  the  combination  of  SS  interference  and  non-Gaussian  impulsive  noise. 
In  [9],  adaptive  receivers  are  proposed  that  are  comprised  by  a  vector  of  adaptive  chip-based 
non-linearities  followed  by  an  adaptive  linear  tap-weight  filter.  The  structures  proposed  in[8]- 
[9]  tap  the  relative  merits  of  both  non-linear  and  linear  signal  processing  and  exhibit  superior 
BER  performance  in  the  presence  of  combined  impulsive  and  SS  interference.  In  particular,  the 
non-linear  receiver  front-end  adapts  itself  to  the  unknown  prevailing  noise  environment  provid¬ 
ing  robust  performance  over  a  wide  range  of  underlying  noise  distributions,  while  the  adaptive 
linear  tap-weight  filter  that  follows  the  non-linearly  processed  chip-samples  combats  effectively 
the  SS  interference.  This  paper  extends  the  work  in  [8]-[9]  in  the  following  aspects.  It  shifts 
the  receiver  design  objective  to  superior  bit  error  rate  (BER)  performance  under  rapid,  short- 
data-record  adaptation  and  the  system  model  is  generalized  to  account  for  multipath  signal 
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reception.  An  additional  non-linear  Hampel-type  pre-processor  is  considered  that  combines  the 
characteristics  of  the  non-linearities  studied  in  [8]- [9]. 


II.  System  Model 


Our  system  model  involves  a  signal  of  interest  with  code  S0  of  length  L  chips  (if  T  is  the 
symbol  period  and  Tc  is  the  chip  period  then  L  =  T/Tc ),  M  multipaths,  (K-l)  multipathed  DS- 

SS  interferes  with  signatures  Sk,  k  =  1, - A  -  1,  and  non-Gaussian  (impulsive)  interference. 

For  notational  simplicity  and  without  loss  of  generality  we  choose  a  synchronous  set-up.  We 
assume  that  the  multipath  spread  is  of  the  order  of  a  few  chip  intervals  and  since  the  signal  is 
bandlimited  to  B  =  1  /Tc,  the  tap-delay  line  channel  model  has  taps  spaced  at  chip  intervals 
Tc.  The  chip-synchronous  received  signal  vector  during  a  symbol  interval  that  results  after 


conventional  chip-matched  filtering  and  sampling  at  the  chip  rate  comprises  of  L  +  A/  -  1 
samples  and  is  given  by 


K- 1  M 

F  =  Ck,m\JEkSktm  +  n,  (1) 

k= 0  m= 1 

where,  with  respect  to  the  £-th  SS  signal.  bk  and  Ek  are  the  transmitted  bit  and  energy, 
respectively,  and  {c*..m}  are  the  coefficients  of  the  frequency-selective  slowly  fading  channel 
modeled  as  independent  zero-mean  complex  Gaussian  random  variables  that  remain  constant 
within  a  bit  interval.  Sk,m  is  the  signature  of  the  k- th  SS  signal  corresponding  to  its  n?-th 
path  (we  assume  0-padding  for  the  expansion  of  Sfc  to  S*.m)  and  n  is  assumed  to  be  complex 
non-Gaussian  impulsive  noise. 

An  equivalent  representation  of  the  received  signal  r  is  given  by 


r  =  V&0w£>mp  +  I  +  n. 


(2) 
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where  w^0)MF  =  J2m= 1  co.mSo.m  is  the  effective  channel  processed  signature  of  the  SS  signal  of 
interest  (signal- 0).  In  (2),  I  identifies  comprehensively  both  the  Inter-Symbol  and  the  SS  inter¬ 
ference.  We  use  the  subscript  R-MF  in  the  effective  signature  notation  to  make  an  association 
with  the  RAKE  Matched- Filter  receiver  that  is  known  to  correlate  the  signature  S0  with  M 
siz e-L  shifted  windows  of  the  received  signal  (that  correspond  to  the  M  paths  of  the  channel) 
appropriately  weighted  by  the  conjugated  channel  coefficients,  co,mim  =  1,  •  •  •  •  M. 

III.  Distribution-free  non-linear  pre-processors. 

In  this  section  we  generalize  the  distribution-free  non-linearities  considered  in  [8]- [9]  for  the 
multipath  model  adopted  and  we  also  propose  to  study  an  additional  Hampel-type  non-linearity. 
The  non-linearities  used  in  the  present  work  are  defined  below: 

Clipper-type  non-linearity 

.  x,  if  Ixl  <  c, c  >  0 

9i(x)=<  (3) 

cf- T,  if  Ixl  >  c,  c  >  0. 

^  Fl 

Puncher-type  non-linearity 

x,  if  Ixl  <  c,  c  >  0 

(4) 

0,  if  jx |  >  c,  c  >  0. 

Hampel-type  non-linearity 

x,  if  jx |  <  a,  a  >  0 

aA,  if  a  <  jx |  <  6,  a,  b  >  0 

<?3(*)=  W'  (5) 

c~J^Iqt^7.  if  b  <  jx |  <  c,  a,  6,  c  >  0 


0, 


otherwise  . 


In  all  the  above  non-linearities,  x  is  a  complex  number  and  |a:|  denotes  the  norm  of  x.  The 
linear  region  of  the  non-linearities  has  the  effect  of  passing  the  observations  undistorted.  The 
non-linear  regions  either  completely  reject  (remove)  or  “correct”  the  observation.  The  latter 
is  considered  as  an  adjustment  of  the  norm  while  maintaining  the  phase.  The  parameters 
a.  b,  and  c  that  appear  in  the  above  non-linearities  are  positive  cutoff  parameters  to  be  deter- 
mined  adaptively. 


IV.  MS-type  linear  post-processing 

1.  Non-multipath  case 

To  combine  the  best  features  of  linear  filtering  and  non-linear  signal  processing,  we  consider 
an  adaptive  non-linear  receiver  (Figure  1)  for  the  detection  of  the  information  bits  of  a  spread- 
spectrum  signal  of  interest  in  a  non-Gaussian  impulsive  noise  environment. 


Figure  1:  General  Structure  of  a  Non-Linear  DS-CDMA  Receiver 
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The  receiver  involves  a  non-linear  pre-processing  front-end  followed  by  a  linear  tap-weight 
filter.  The  non-linearities  are  in  general  non-identical.  As  discussed  [8]-[9],  approximately  opti¬ 
mum  performance  can  be  obtained  if  the  non-linearities  are  chosen  on  the  basis  of  explicit  apriori 
knowledge  of  the  underlying  combined-noise  statistics.  In  realistic  applications,  however,  this 
information  is  not  usually  available  and  our  approach  proceeds  by  choosing  a  parametrically 
described  distribution  free  non-linearity  which  adapts  its  parameters  to  the  unknown  prevailing 
noise  environment.  This  way  we  develop  a  receiver  that  can  be  efficiently  optimized  and  ex¬ 
hibits  robust  performance  over  a  wide  range  of  underlying  noise  distributions.  In  the  following, 
the  non-multipath  case  is  presented  in  the  context  of  either  the  clipper  or  the  puncher.  These 
non-linearities  are  adaptive  with  respect  to  the  cutoff  parameter  as  shown  in  Figure  2. 


Figure  2:  (a)  Clipper,  (b)  Puncher. 

The  overall  system  parameters  are  determined  by  either  a  joint  MSE-BER  optimization  al¬ 
gorithm,  or  a  joint  BER  algotithm  or  a  joint  MSE  algorithm,  as  described  below. 

A.  Joint  MSE-BER  Optimization 

In  this  approach  the  non-linearly  processed  chip  samples  drive  a  linear  tap-weight  filter  whose 
taps  are  determined  according  to  the  MMSE  or  MVDR  criterion.  Under  both  criteria  the  tap- 
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weight  filters  are  determined  by  the  statistics  of  the  non-linearly  processed  received  signal  sam¬ 
ples.  Let  g(-)  =  [<7i(-), . . . ,  <7l(-)]  denote  the  vector  of  the  non-linearities  and  wg  =  [u’i, . . . ,  wl\t 
denote  the  linear  tap-weight  filter  parameterized  by  the  non-linearity  g(-).  Then  the  test  statis¬ 
tic  for  the  detection  of  the  information  bit  of  the  spread-spectrum  sign  al  of  interest  is  deter¬ 
mined  by  the  inner  product 

<  g(r)>wg  >  (6) 

and  appropriate  tap-weight  vectors  wg  are  given  by  the  following  proposition. 


Proposition  1  Given  the  non-linearity  g(-)  then: 

(i)  the  MMSE  linear  tap-weight  vector  wg  is  given  by 

Wg,MMSE  =  Rg-1jB{g(r)60}  .  (7) 


(ii)  the  MVDR  linear  tap-weight  vector  wg  that  is  distortionless  in  the  S0  direction  is  given  by 


Wg.MVDR 


1 


Try  -lo  ^8  ^°  • 


So  Rg  So 

For  both  (i)  and  (ii)  the  covariance  matrix  R  is  given  by 


(8) 


Rg  =  £{g(r)gT(r)}-  (9) 

Proof 

(i)  The  filter  w  in  (7)  is  the  Wiener  solution  for  a  linear  tap- weight  filter  with  input  vector 

g(r). 

(ii)  The  filter  w  in  (8)  is  the  solution  to  the  optimization  problem  that  minimizes  the  output 
variance  under  the  constraint  that  wtSq  =  1.  □ 


Corollary  1  If  the  non-linearity  gff)  is  the  clipper  or  the  puncher  with  cutoff  parameter  equal 
to  c.  then: 
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(i)  the  MMSE  linear  tap-weight  vector  is  given  by 

Wc.MMSE  =  Z^iR^So,  (10) 

(ii)  the  MV  DR  linear  tap-weight  vector  is  given  by 

WcMVDR  =  c  rp-ic  Rc~ls°-  U1) 

I>0  H-c  ^0 

For  both  (i)  and  (ii)  R  c  is  given  by  (32)  with  g(-)  being  the  vector  of  non-linearities  denoted  as 
gc(-)  and  3\  is  equal  to  the  probability  that  —c  <  r,-  <  c. 

Proof 

Let  8i.  and  83  denote  the  probabilities  that  r,  6  [— c,  c],  r,  >  c,  and  r,  <  — c,  respectively.  If 
£;(•)  is  the  clipper  then  E{gi(ri)b0}  =  8iE{r,b0}  +  /32£{c£>o}  +  ^^{-cfto}  =  If  ^ ( • ) 

is  the  puncher  then  E{gi(ri)bo}  =  ft\E{ribo}.  Part  (ii)  can  be  proven  similarly.  □ 

The  filters  given  in  ( i )  and  (ii)  of  Corollary  1  result  in  equivalent  receiver  structures  since 
the  decision  operation  sgn(<  gc(r),wc  >)  is  positive-scalar  inv  riant. 

The  simplicity  of  the  particular  parametric  non-linearities  (clipper  and  puncher)  together 
with  the  adaptive  evaluation  of  the  cutoff  parameter  result  in  a  simple  structure  that  exhibits  the 
favorable  characteristics  of  both  conventional  linear  filtering  and  non-linear  signal  processing. 
The  cutoff  parameter  c  is  related  to  our  confidence  on  the  interval  [— c.  c]  and  the  fraction  of  the 
received  data  that  we  expect  to  fall  in  the  linear  region  of  the  receiver  non-linearity.  Thus,  in 
highly  impulsive  environments  a  reasonably  small  value  of  the  cutoff  parameter  c  will  prohibit 
the  contamination  of  the  decision  statistic  (i.e.  the  output  of  the  linear  filter  post-processor) 
by  high  amplitude  disturbances.  On  the  other  hand,  the  more  Gaussian  (non-impulsive)  the 
environment  is,  the  larger  the  cut-off  parameter  value  can  be  and  the  closer  to  the  conventional 
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linear  MMSE  or  MVDR  filter  the  proposed  receiver  structure  is.  In  other  words,  the  cutoff 
parameter  c  tracks  the  impulsiveness  of  the  environment  and  the  overall  structure  converges, 
as  c  — >  oo,  to  the  conventional  linear  receiver.  For  the  evaluation  of  the  parameter  c  let  us 
assume  without  loss  of  generality  that  the  receiver  employs  the  same  non-linearity  per  chip 
sample  gc(-)  parameterized  by  the  cutoff  parameter  c.  Let  us  then  denote  the  output  of  the 
receiver  by  u(r,c)  =  sgn  <  gc(r).  wc  >.  If  we  define  />(■)  such  that 

p(r0,  Ti‘.  c)  =  ~ { 7r0[l  +  u(r0.c)]  +  jra[l  -«(n,c)]},  (12) 

where  tt0  =  =  1/2  are  the  a  priori  probabilities  of  hypothesis  H0  and  Hu  then  we  observe 

that  the  probability  of  error  is  equal  to 


Pe{c)  =  £{p(r0,ri;c)} ,  (13) 

where  r0  and  ri  correspond  to  received  data  vectors  from  H0  ( b0  =  -1)  and  Hi  (6j  =  +1) 
respectively.  Thus,  the  so  defined  function  p(-)  (cf.  (12))  provides  a  measure  for  the  distortion 
at  the  output  of  the  receiver.  Indeed,  the  bracket  terms  are  both  zero  when  the  receiver  makes 
the  correct  decision  and  strictly  positive  otherwise.  Based  on  stochastic  gradient  techniques 
and  exploiting  the  property  in  (13).  we  develop  an  adaptive  procedure  that  iteratively  adjusts 
the  cutoff  parameter  c  to  minimize  the  probability  of  error  at  the  output  of  the  receiver: 


^n+ 1  ^n^n(^n)  i 


(14) 


w 


here 


x«(c)  =  ^-[^(ro,n,ri,n;c  +  dn)-p(r0.„,r1.„;c-dn)], 


(15) 


and  |dn}  and  {q„}  are  monotonically  decreasing  sequences  of  positive  numbers  such  that 
£  On  =  oc,  £  andn  <  oo,  and  £  <  oo. 
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Summarizing,  given  the  non-linearity  gc(-)  and  the  corresponding  cutoff  parameter  c,  con¬ 
ditional  MS  optimization  results  in  the  ideal  filter  given  by  Proposition  1  and/or  Corollary  1. 
Since  the  filter  w  itself  is  a  function  of  c,  further  optimization  with  respect  to  c  according  to 
(14)  isolates  the  filter  that  minimizes  the  probability  of  error  given  by  (13).  We  may  view 
the  overall  optimization  procedure  described  above  as  a  set  of  coupled  equations  comprised  by 
eq.  (14)  and  (8).  To  implement  this  set  of  coupled  equations  we  develop  and  examine  three 
alternative  schemes  which  we  call  MSE-BER  Type  I,  Type  II,  and  Type  III,  respectively. 

a)  MSE-BER  Type  I 

The  (n  +  1)  iteration  step  of  the  algorithm  is  given  below: 

<-n+l  Q:nXn(cn)  ,  (16) 

wcn+1  =  R;n1+l^{gc„+l(r)6o},  (17) 

where  RCn+1  =  £{gc„+,  (r)g^+l  (r)}  .  Each  iteration  step  of  the  algorithm  requires  one  sample 
(per  hypothesis)  for  the  cutoff  parameter  recursion.  On  the  other  hand,  the  quantities  R^1 
and  £{•}  for  the  evaluation  of  wCn+,  are  estimated  by  SMI  (Sample  Matrix  Inversion)  and 
sample  averaging,  respectively,  based  on  the  whole  data  record. 

b)  MSE-BER  Type  II 

The  (n  +  1)  iteration  step  of  the  algorithm  is  still  given  by  ( 16)-(  17)  as  in  Type  I  above.  However, 
in  this  case,  at  each  iteration  step  of  the  algorithm  a  data  block  of  increasing  size  is  used.  In 
particular,  the  first  sample  (per  hypothesis)  is  used  for  the  recursion  of  the  cutoff  parameter 
while  the  rest  are  used  in  the  evaluation  of  wCn+1  for  SMI  and  sample  averaging  estimation  of 
RCn+1  and  £{•}.  respectively.  In  comparison  with  Type  I,  Type  II  implementation  of  the  joint 
MSE-BER  algorithm  utilizes  fewer  data  for  the  estimation  of  wCn+1  per  iteration  step.  This 
implies  that  Type  II  is  less  computational  intensive  but  is  slower  and  performs  worse  than  Type 
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I  in  the  early  steps  of  the  algorithm, 
c)  MSE-BER  Type  III 


To  improve  the  initial  BER  of  the  MSE-BER  Type  II  coupled  recursions  we  develop  the  Type 
III  implementation  that  proceeds  according  to  the  following  recursions: 

Cn+\  Cn  OtnXn(cn)  ,  (  IS) 

wCn+1  =  ZMS(initial  value  w<°^, data  block  size  An+1, LMS  step  size,  cn+i)  (19) 

where  {An}  denotes  the  sequence  of  increasing  data  block  sizes  and  LMS(>, denotes  an 
LMS  loop  for  the  evaluation  of  the  filter  in  the  (n+l)-iteration  step  of  the  algorithm  using  a 
data  block  of  size  An^_j.  The  final  output  of  the  LMS  loop  is  where  the  superscript 

denotes  the  internal  LMS  iteration  index.  The  initial  value  of  the  filter  in  the  LMS  loop  is 
chosen  to  be  equal  to  i.e.  w<°>+1  =  We  also  indicate  the  cutoff  parameter  cn+i  as 

well  as  the  LMS  step  size  as  additional  parameters  of  the  LMS  loop. 

The  receivers  developed  so  far  use  different  criteria  for  the  optimization  of  the  cutoff  param¬ 
eter  and  the  linear  filter  (minimum  BER  and  MMSE  respectively).  Below  we  propose,  develop, 
and  stud}/  two  alternative  optimization  approaches  that  make  use  of  the  same  criterion  for  joint 
optimization  of  the  overall  system  parameters. 

B.  Joint  BER  Optimization 

Let  a  denote  the  combined  vector  of  the  adjustable  receiver  parameters,  i.e.  the  filter  tap- 
weights  and  the  cutoff  parameter  c  of  the  non-linearity  (clipper  or  puncher).  That  is. 

ar  =  [oi,a2,...,aL+1]  =  [w  T,c]  =  [t^,  uj2,  . . .  wL,  c] ,  (20) 

\\  here  w  is  the  filter  tap-weight  vector.  Then  the  operation  performed  by  the  non-linear  receiver 
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on  the  received  vector  r  parameterized  by  the  combined  vector  a  can  be  defined  as 

W(r,a)  =  sgn  <  yOL+1(r),w  >  .  (21) 

(i)  Clipper  Non-Linearity 
We  define  Qnp(-)  such  that 

Cc/iP(ro,ri;a)  =  7r0/co[W(ro,  a)]  +  jra/cl[W(  r^a)]  (22) 

where  7To  and  7Tj  are  the  a  priori  probabilities  of  Ho  and  Hi  respectively,  while  /c0(;r)  =  |(1  +  x) 
and  fc j(-)  =  |(l  —  x).  Then  the  probability  of  error  is  equal  to 

Pe(a)  =  £{Cci.>(r0,ri;a)}  (23) 

and  the  vector  a  that  minimizes  (23)  can  be  found  using  the  recursion 

an+i  =  an  -  a„y„(a„),  (24) 

where 

yn  =  [yn,i(a),...,y„.L+i(a)]  and  (25) 


ynj(a)  —  n  ,  [Cci»p(^"o,n ?  ri,n>  a  T  dnGj )  CcHp(t*o,n)  a  dnej)].  (26) 

In  the  above  expressions,  ej  is  the  j  —  th  coordinate  unit  vect  or  while  { dn }  and  {an}  are 
monotonically  decreasing  sequences  of  positive  numbers  such  that  =  oc,  ]Tandn  <  oc. 


and  Y2  andn2  <  °°- 


(ii)  Puncher  Non-Linearity 

In  this  case,  the  possible  receiver  outputs  are  0,  +1,  —1.  HU(r,  a)  is  0,  then  we  decide  Hi  w.p. 
~i  and  Hq  w.p.  tq.  Thus,  under  either  hypothesis  the  probability  of  error  is 


P(e\H0)  =  P[W(r0,a)  =  1]  +  7riP[W(r0,a)  =  0]  and 


P{e\Hi)  =  P[ZY(r!,a)  = -1]  +  7r0P[W(ri,a)  =  0], 


(27) 


If  we  define  the  functions  /P0(-)  and  /P1(-)  as 


/po^)  —  i 


'pi' 

0,  x  =  —  1 

t n,  i  =  o  and  /pi(^)  = 

1,  x  =  +1 


1,  X  —  —  1 
7T0,  X  =  0 
0,  X  =  +1 


and  CPuncA(r0,iV,a)  =  7r0/P0(^(r0,  a))  +  7r1/P1(^(r1,  a)) , 


(28) 


(29) 


then 


E{Qunch{ ro.rx:a)}  =  7ro{0  ■  P[U( r0,a)  =  -1]  +  1  •  P[^(r0,a)  =  1]  +  jti  •  P[W(r0,a)  =  0]}  + 

tti {0  •  P[U{ ri,a)  =  1]  +  1  •  P[W(ri,a)  =  -1]  +  tt0  •  P[U{ruSi)  =  0]} 

=  TrQP(e\H0) +  irlP{e\Hi)  =  P{e).  (30) 

and  the  same  recursion  as  in  (24)  with  (ptm c/,(-)  in  place  of  ( clip(‘)  yields  the  appropriate  vector 
a.  We  note  that  when  ttj  =  tt0  =  |  then  fc0(-)  =  /P0(-)  and  /cl(-)  =  /P1(-),  which  implies  that 

C  clipi")  Cpunc/i(')- 


C.  Joint  MSE  Optimization 

(i)  Clipper  Non-Linearity 

We  recall  that  aT  =  [wT,  c]  =  [uq, . . . ,  wi+ j,  cj.  Then  the  MSE  between  the  output  of  the  linear 
filter  and  a  desired  bit  sequence  of  the  spread-spectrum  signal  of  interest  is  given  by 

J  =  E{«w,gc(r)>  -b0)2}  =  E{ctwi9c(rt)  -  b0)2}  (31) 

t=i 

and  the  MS-adaptive  implementation  for  the  combined  parameter  vector  a  is  given  by  the 
following  recursion: 

a(n  +  1)  =  a(n)  —  gV(J)(n),  (32) 
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where  y  is  the  constant  step-size  parameter  and  V  denotes  the  instantaneous  estimate  of  the 


gradient;  that  is 

V(J)  =  (V,(J) . VL(,J),VL+l(J)].  (33) 

A 

V,(J)  =  JL(J)  =  2(YlV’i9c(rt)-b0)gc(r,).  7  =  1,...  I,  (34) 

A 

Vl+i(J)  =  ^;(J)  =  2(%2wigc(ri)  -  fe0) (35) 
Also,  since  <7C(-)  is  the  clipper,  we  can  write: 

9c{ri)  =  rt[r]{rt  +  c)  -  77(7-;  -  c)]  +  cq(rt  -  c)  -  cr](-ri  -  c),  (36) 

Q 

Q-C9c(ri)  =  7?(r.  -  c)  -  T7(-r,-  -  c),  (37) 

where  ?/(•)  is  the  unit-step  function,  and  c  is  a  positive  scalar. 


(iij  Puncher  Non-Linearity 

Since  the  partial  derivative  with  respect  to  the  cutoff  parameter  c  of  the  MSE  at  the  output 
of  a  system  that  involves  the  puncher  non-linearity  is  zero  almost  everywhere,  joint  MSE  opti¬ 
mization  is  not  feasible  in  this  case. 


2.  Multipath  case 

A  solution  for  the  suppression  of  SS-interference,  intersymbol  interference  and  non-Gaussian 
(impulsive)  noise  that  assumes  the  least  information  about  these  signal  disturbances  in  the  pre¬ 
sense  of  multipath  (i.e.  only  the  effective  code  of  the  signal  of  interest  is  assumed  known)  can  be 
obtained  by  an  adaptive  linear  filter  designed  according  to  a  Minimum  Variance  Distortionless 
Response  (MVDR)  criterion  preceded  by  a  vector  of  adaptive  chip-based  non-linearities.  This 


2-15 


MVDR-tvpe  receiver  is  a  generalization  of  the  receiver  in  [8]-(9]  for  the  multipath  case.  The 
linear  post-processor  is  a  RAKE-type  filter  that  minimizes  the  output  variance  subject  to  the 
constraint  that  the  filter  remains  distortionless  in  the  normalized  “effective”  direction/signature 


wII°r.mf|i  the  user  of  interest.  It  is  given  by 


R  -1w<0’ 

”•8  W||r-mf|| 


g.R-MVDR  <  0)H 


HlR-Mpll^g  W||r-mf|| 


where  Rg  denotes  the  autocorrelation  matrix  of  the  size  (L  -f  M  —  1)  non-linearly  processed 
received  signal  that  includes  all  M  resolvable  paths  ( L  is  the  signature  length),  i.e.  Rg  = 
£{g(r)gff(r)}.  To  form,  however,  the  MS-optimum  adaptive  filter  w<°^MVDR,  exact  knowledge 
of  the  size  [L  +  M  -  1)  data  autocorrelation  matrix  Rg  is  required.  Since  Rg  is  unknown 
we  may  use  available  non-linearly  processed  received  data  to  form  a  sample-average  estimate 
Rg  =  g(rn)gff(r„)  and  then  proceed  with  Sample-Matrix-Inversion  (SMI).  In  this  case 


r>  -1 

Hg  WI1*-mf|| 

w(,0)"  R-iw'01 

llR-MFll  8  W|IR-MFII 


In  the  above  expressions,  the  normalized  effective  signature  is  given  by 


Finally,  the  decisions  are  made  according  to 


b0  =  sgn(Re[  w;°>"MVDRg(r)]), 


where  sgn(-)  denotes  the  sign  operation. 


V.  Proposed  algorithmic  developments 

It  is  known  that  batch  SMI  adaptive  procedures  outperform  recursive  constraint-LMS  adaptive 
implementations  of  the  MVDR  solution  in  terms  of  small-sample  convergence  characteristics. 
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Apart  from  the  computational  complexity  of  the  matrix  inversion  operation,  N  >  L  +  M  -  1 
non-linearly  processed  data  samples  are  required  for  the  estimated  covariance  matrix  Rg  = 
A  Yln=\  g(r)g(r)H  to  be  invertible  with  probability  1  and  in  fact  data  sizes  many  times  the 
filter  length  ( L  +  M  -  1)  are  necessary  for  w‘°^MVDR  =  K  =  [wJ^R" 1  w^0)MF] - 1 ,  to 

approach  reasonably  well  the  performance  characteristics  of  the  “ideal”  MVDR  filter,  where 
“ideal”  refers  to  the  use  of  an  asymptotically  large  data  record  (perfectly  known  Rg).  Unfor¬ 
tunately,  for  typical  L  and  M  values  and  data  transmission  rates,  filter  adaptation  over  that 
many  symbol  intervals  may  not  keep  up  with  typical  channel  fluctuation  (fading)  rates.  This 
discussion  brings  us  to  the  core  concept  of  this  section.  That  is  to  develop  an  SS-receiver  that 
(i)  operates  in  the  presence  of  both  SS  and  impulsive  interference,  (ii)  taps  the  merits  of  both 
non-linear  signal  processing  and  adaptive  linear  filtering,  (iii)  exhibits  inherently  low  compu¬ 
tational  complexity  and,  at  the  same  time,  (iv)  maintains  superior  short-data-record  filtering 
performance. 

To  gain  some  insight  with  respect  to  the  proposed  algorithmic  framework,  we  consider  a 
canonical  representation  of  the  linear  filter  post-processor  as  depicted  in  Figure  3. 


Figure  3:  Proposed  receiver  structure. 

In  this  figure  we  look  at  the  class  of  scalar  parameterized  FIR  linear  post-processing  filters  of 


the  form 


Wprop  W||r-mf||  AtgGg’  (42) 

where  /ig  is  an  arbitrary  complex-valued  parameter,  and  Gg  is  an  arbitrary  “auxiliary”  complex 
vector  of  dimension  L  +  M  -  1  that  is  orthonormal  with  respect  to  w;,0)  „  =  w(0)  /llw'0’  II- 

~  j|R-MF||  R-MF/  II  R-MFll* 

Ggw||R-MF||  =  0  and  || Gg ||  =  1.  (43) 

Both  fi g  and  Gg  are  to  be  designed.  Bit  detection  is  made  as  follows: 

60  =  s5n(i?e[w'°>"pg(r)]).  (44) 


We  note  that  both  Gg  and  /rg  are  parametrized  by  the  f  x  1  vector  of  non-linearities  g(-)  and 
for  any  Gg  satisfying  the  constraints  in  (43),  the  filter  wj|’_MF||-^gGg  is  already  “distortionless” 
in  the  wf£  MF||  direction  of  interest  since  mf(|  =  1.  For  an  arbitrary  but  fixed  auxiliary 

vector  Gg.  the  filter  in  (42)  can  be  viewed  as  scalar  parameterized  and  optimized  with  respect 
to  the  complex  scalar  /xg  only.  The  MS-optimum  value  of  /rg  is  the  value  that  minimizes  the 
MSE  £{|w||TMF||g(r)  -  /'gGfgO*)!2}  between  points  (a)  and  (c)  in  the  block  diagram  filter 
representation  in  Figure  3. 

The  solution  is  simply 


= 


Gg"RgW||R-MF|| 

(VRgGg 


(45) 


where  Rg  =  £{g(r)g(r)"}. 

Now.  we  return  to  the  problem  of  selecting  an  auxiliary  vector  subject  to  an  appropriately 
chosen  criterion.  The  selection  criterion  for  the  auxiliary  vector  Gg  that  we  propose  to  study 
in  this  paper  is  motivated  by  the  MSE  interpretation  of  the  filter  in  Figure  3  and  is  the  max¬ 
imization  of  the  magnitude  of  the  cross-correlation  between  points  (a)  and  (b)  subject  to  the 
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constraints  in  (43): 


Gg  =  argrnax|£;{w[10I(HMF||g(r)(GHg(r))*}| 

=  arS  max  |wg"MF||RgG| ,  (46) 

subject  to  G^ mf||  =  0  and  Gg  Gg  =  1 . 

The  magnitude  cross-correlation  criterion  function  |w||°,l.MF||R-gGg|  an<4  the  orthonormality 
constraints  are  all  phase  invariant.  So,  to  avoid  unnecessary  ambiguities  and  without  loss 
of  generality,  we  can  identify  the  unique  auxiliary  vector  that  is  a  solution  to  our  constraint 
optimization  problem  and  makes  w||0RWMF||RgGg  real  non-negative  (w|l|0RWMF||RgGg  >  0).  This  is 
the  vector 

C  _  ~  1W||r-Mp||^W||«UMP||JW||r-MF|| 

8  llRgW||B-MF||  "  ( W&F||RgW||R.MF||)W||LF||  II  ' 

We  may  comment  at  this  point  that  the  fact  that  no  matrix  inversion  is  explicitly  required  or 
implicitly  assumed  is  expected  to  result  in  superior  performance  in  realistic  small-data-record 
adaptive  operation. 

The  Max-Cross-Correlation  (MCC)  approach  can  be  generalized  for  processing  with  multiple 
auxiliary  vectors.  In  particular,  we  propose  to  consider  conditional  optimization,  i.e.  to  deter¬ 
mine  the  new  vector,  say  G2,  and  the  new  scalar,  say  /i2,  by  assuming  that  the  main  direction 
branch  wj^  MF||  (Figure  3)  has  been  updated  and  fixed  to  MF||  —  pi Gj,  where  Gi  and  pi  were 
determined  by  (47)  and  (45).  These  general  algorithmic  developments  are  summarized  in  the 
following  Proposition. 

Proposition  2  Given  the  nonlinearity  g(*)  then: 

(i)  The  maximum  cross-correlation  linear  tap-weight  filter  w^OP  is  given  by 


w 


(0) 

PROP 


A 


W 


(0) 

||r-mF|| 


M  gGg, 


(48) 
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where 


H  = 


and 


Gg  - 


R„w 


(0) 


g^vC.,, 

G"RgGg 


(0) 


8  ||r.mp|| 


r(0) 


(49) 


(50) 


IP  wp(^)  _  fw(°)W  P  w(°)  ^w(0)  il  * 

ii  8  ||r-mf||  v  ||r-mf||  8  ||r-mf||/  || Ft-MF*||  I ! 

(ii)  The  generalized  multiple  auxiliary  vector  linear  post-processor  is  given  by  (l  <  P  <  L  + 
M  -  1 ) 


w 


PROP-Mul 


w 


(0) 


i=i 


where  {G,}  and  {^,}  are  determined  recursively  as  follows: 

Rt(wfll,ll-EL,ftG,)- 


Gp+1  = 


R.(w|,1hi  -  Ef=i  ft  G.) 


(51) 


r-n7(0)H  R  /w'U|  VP  „  r  Mur'0' 

lW||R-MF||K'g(W||R-MFll  _  2-.  =  l  /i>'(j'')JW||R.MF||  “ 


(0) 


["N°1L|RB(wr„ll  -  el,  f,g,)]w™ 


f(°) 


(0) 


and 


-  s;„[GfRii(W|i°|M,||  -  el,  ftgoigj 

-  EJ.,[GfR,(wJ0>  -  EL,  ftG,)]Gj||  ’ 


G^.Rjlwjlrti-EL.F.G,; 

G^+iRgGp+i 


/^P+  1 


(52) 


(53) 


/or  1  <  p  <  £  +  M  —  2,  and  Gi,  /ij  given  by  (50)  and  (49),  respectively.  □ 


VI.  Simulation  Studies 

The  optimization  of  the  filter  parameters  follow  the  general  principles  of  MSE-BER  (Type  I) 
optimization  of  Section  IV.  1  The  corresponding  algorithm  is  generalized  below  for  the  multipath 
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case.  In  particular,  let  the  receiver  output  be  denoted  by  u( r,  c),  i.e.  u(r,  c)  =  sgn(Re[ w^gc(r)]) 
where  c  is  the  cutoff  parameter  of  the  non-linearity  involved  (eg.  puncher  or  clipper  type).  Then, 
the  distortion  measure  p(ro,ri;c)  defined  at  the  receiver  output  by 

p(ro,rx;c)  =  ^{7r0[l  +  u(r0,c)]  +  jtx[1  -  u(rx,c)]}  ,  (54) 

where  7To  =  ttx  =  1/2  are  the  a  priori  probabilities  of  hypothesis  Ho  (bo  =  —  1)  and  H\  (bo  =  +1). 
attains  an  average  value  equal  to  the  BER,  i.e. 

Pe(c)  =  £{/!)(ro,ri;c)}  ,  (55) 

where  r0  and  rx  correspond  to  received  data  vectors  from  H0  and  H\  respectively.  Exploiting 
the  property  in  (55),  we  can  develop  a  stochastic  gradient  technique  that  adjusts  the  cutoff 
parameter  c  to  minimize  the  probability  of  error  at  the  output  of  the  receiver,  as  follows: 


Cn+l  —  ^n®n(Cn)  i 


where 

xn(c)  =  ^Vb(ro.n,ri,n;c  + dn)  -  p(r0,„,rx,n;c- dn)] ,  (57) 

2dn 

and  {dn}  and  {crn}  are  monotonically  decreasing  sequences  of  positive  numbers  such  that 
£3  On  =  oc,  Ylandn  <  oo,  and  ')2a2nd~2  <  oo.  A  similar  gradient  alogrithm  can  be  obtained 
when  more  than  one  cutoff  parameters  are  involved  in  the  non-linearity  used  (eg.  Hampel-type). 

We  examine  DS-SS  transmissions  in  the  presence  of  5  SS  interfering  signals  and  impulsive 
noise.  The  normalized  signature  cross-correlation  of  the  interfering  signals  with  the  signal  of 
interest  is  approximately  25%  and  the  interfering  signal  energies  are  equal  to  6,  7,  8.  9.  lOdB. 
The  system  processing  gain  is  63.  Figure  4  emphasizes  the  BER  performance  of  the  proposed 
receiver  (implemented  with  one  and  two  auxiliary  vectors)  and  the  MVDR  receiver  as  a  function 


of  the  norm  of  the  projection  of  the  SS  signal  of  interest  into  the  interfering  subspace,  when 
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the  clipper-type  or  puncher-type  non-linear  processing  front-end  is  employed.  Receiver  needed 
ensemble  averages  are  replaced  by  sample- average  estimates  based  on  a  data-record  of  150 
samples.  The  channel  noise  is  modeled  according  to  the  the  outlier  model,  described  in  the 
following  paragraph  with  the  difference  that  no  multipath  is  considered  for  this  figure  (ie.  all 
functions  are  real). 

The  multipath  fading  channel  is  modeled  as  having  4  paths  with  zero  mean  complex  Gaussian 
fading  coefficients  of  variance  0.5  (i.e.  £{|c*,m|2}  =  0.5)  for  all  paths  and  SS  signals.  The 
impulsive  channel  noise  is  modeled  according  to  the  e-mixture  disturbance  model  given  bv  the 
following  expression: 

Mx)  =  (1  -  f)/o(-r)  +  e/i(x) .  (58) 

where  e  €  [0,1]  accounts  for  the  probability  under  which  the  noise  is  fx{-)  distributed.  The 
nominal  pdf  /0(-)  is  taken  to  be  complex  Gaussian  with  variance  <?%  =  !.  The  ‘'contaminating'’ 
P^f  /i(-)  is  assumed  to  be  either  a  complex  Gaussian  pdf  with  variance  a\  =  ~2a^  (-y2  =  1.000) 
or  a  delta  function  at  ±oo  ±  joo.  The  resulting  model  is  the  Gaussian  e-mixture  model  and 
the  outlier  model,  respectively. 

We  compare  the  BER  performance  of  the  proposed  receiver  (implemented  with  one  and 
two  auxiliary  vectors)  with  that  of  the  MVDR  receiver  when  the  clipper-type.  Hampel-type,  or 
puncher-type  non-linear  processing  front-end  is  employed.  The  BER  of  the  conventional  MVDR 
filter  is  also  included  as  a  reference  point.  The  results  are  averages  over  100  independent  channel 
realizations. 

In  Figure  5  we  present  the  BER  performance  of  all  receivers  as  a  function  of  the  energy 
of  the  SS  signal  of  interest  for  the  Gaussian  e-mixture  model.  Figure  6  carries  out  the  same 
studies  for  the  outlier  model.  For  both  figures,  receiver  needed  ensemble  averages  are  replaced 
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by  sample-average  estimates  based  on  a  data-record  of  150  samples.  Figures  7  and  8  plot  the 
BER  as  a  function  of  the  size  of  the  data  record  for  the  same  set-up  as  in  Figures  5  and  6, 
respectively.  That  is,  Figure  7  shows  the  BER  performance  for  the  Gaussian  e-mixture  model, 
while  Figure  8  deals  with  the  outlier  model. 
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Bit  Error  Rate 
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Figure  4:  Bit  error  rate  as  a  function  of  the  norm  of  the  projection  of  the  SS  signal  of  interest 
into  the  interfering  subspace  in  the  presence  of  outlier  noise. 
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Figure  5:  Bit  error  rate  as  a  function  of  the  energy  of  the  SS  signal  of  interest  in  the  presence 
of  e-mixture  noise. 
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Bit  Error  Rate 


Bit  Error  Rate 


500  1000  1500  2000  2500  3000  3500  4000  4500  5000 

Data  Record  Size 


Figure  8:  Bit  error  rate  as  a  function  of  the  training  set  size  in  the  presence  of  outlier  noise 
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HIERARCHICAL-  ADAPTIVE  IMAGE  SEGMENTATION 


N.  Bourbakis 
Professor. 

Dept,  of  Electrical  Engineering 
Image-Video  Processing  &  Machine  Vision  Lab, 

Abstract 

This  paper  presents  a  software  methodology  for  near  real-time  and  real-time  user 
driven  segmentation  of  color  images  or  sequence  of  images  (video)  by  using  a  low  cost 
approach  (a  low  cost  camera  and  a  PC).  The  segmentation  approach  used  here  is  based 
on  fuzzy-like  reasoning  and  a  color  reflection.  The  methodology  provides  a  human-like 
perception  modeling  in  combination  with  fast  smoothing,  region  growing  and  edge 
detection.  Results  on  several  images  are  provided  to  demonstrate  the  methodology’s 
adaptive  segmentation  capabilities  based  on  the  user’s  requests. 


Keywords :  Segmentation  of  Color  Images,  Adaptive  and  Real-time  Segmentation  of  Sequence  of  images 
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HIERARCHICAL-  ADAPTIVE  IMAGE  SEGMENTATION 


1.  INTRODUCTION 

Segmentation  is  one  the  most  important  processing  or  preprocessing  step  required  by 
almost  any  image  processing  and  vision  task  performed  by  a  machine  [1-18].  It  is  a 
process  that  has  not  a  perfect  solution  for  an  input  image  if  previous  knowledge  is  not 
available.  Thus,  a  number  of  special  or  general  purpose  approaches  have  been  presented 
by  scientists  and  practitioners  to  provide  solve  problems  in  different  areas,  such  short  or 
long  distance  robotic  vision,  ATR,  visual  inspection  in  production  lines,  pattern 
recognition,  OCR,  document  processing,  etc.  [19-25],  Recently,  research  studies  on 
image  segmentation  make  use  of  previous  knowledge  [26,27],  or  attempted  to  extract 
knowledge  by  segmented  a  great  variety  of  images  under  different  resolution  [28].  The 
effort  of  this  paper  belongs  to  the  latter  category,  where  the  authors  attempt  to  extract 
knowledge  and  classify  it  in  various  categories  and  applications.  More  specifically,  they 
have  developed  a  software  tool-methodology,  able  to  segment  an  image  or  a  sequence  of 
images  in  real-time,  depended  on  the  size  of  the  images,  under  various  conditions  of 
sensitivity  controlled  by  the  user  with  a  variety  of  parameters,  such  as  smoothing,  local 
contrast  deviation,  region  filling,  similarity  deviation,  region  growing.  The  methodology 
runs  on  a  PC  under  Windows  95  or  NT. 

2.  FAST  EDGE  PRESERVING  SMOOTHING 

Smoothing  is  an  important  preprocessing  step  for  any  segmentation  operation  in  that  it 
allows  for  the  reduction  of  noise  within  an  image.  Typically,  areas  of  an  image  perceived 
to  be  a  single  solid  color  may  actually  be  composed  of  pixels  which  span  a  broad  range 
similar  colors.  Thus,  a  smoothing  type  operator,  such  as  a  low  pass  filter,  may  potentially 
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make  these  areas  more  continuous  in  their  color  value  while  lessening  the  effects  of  salt 
and  pepper  noise.  However,  one  problem  with  this  type  of  filtering  or  averaging  has 
always  been  the  destruction  or  distortion  (i.e.  dilation)  of  edges  within  the  image.  A 
more  desirable  approach  is  to  selectively  smooth  the  image  in  a  way  that  will  preserve 
edges  or  areas  of  sharp  contrast  present  within  the  image.  Computation  time  is  also  a 
concern  when  developing  such  an  approach  since  pixel-wise  neighborhood  operations 
performed  on  large  images  can  be  quite  slow,  precluding  their  use  within  real-time 
applications  on  general  purpose  (i.e.  low-cost)  hardware.  These  applications  include  real¬ 
time  video  processing  for  tasks  such  as  target  recognition  or  tracking. 

The  method  presented  here  attempts  to  smooth  an  input  image  in  a  timely  fashion  while 
preserving  the  locations  of  edges  within  the  image.  This  edge  preserving  smooth 
operation  is  accomplished  by  reducing  the  number  or  complexity  of  arithmetic  operations 
performed  on  pixel  neighborhoods  and  by  employing  the  use  of  SEME)  instructions  that 
are  now  common  on  inexpensive  general  purpose  microprocessors  such  as  the  Intel 
Pentium  and  Pentium  II  processors  via  MMX.  The  technique  presented  was  adapted 
from  earlier  work  described  in  [8,9]  which  produced  robust  methods  for  smoothing  and 
segmenting  images  with  disregard  to  computational  complexity  or  time  constraints.  The 
method  is  applied  using  either  7  x  7  or  5  x  5  neighborhood  windows  and  considers  a 
pixel’s  local  contrast  before  deciding  if  or  how  smoothing  is  performed  (see  figure  1). 
Color  contrast  between  any  two  colors  may  be  thought  of  as  a  value  proportional  to  the 
Euclidean  distance  between  those  colors  in  the  RGB  color  space  (illustrated  in  figure  2). 
The  overall  algorithm  is  outlined  in  figure  1 . 
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The  entire  smoothing  algorithm  is  implemented  entirely  in  assembly  language  making 
significant  use  of  the  MMX  instruction  set  present  on  Intel  Pentium  family  processors. 
To  save  time  and  avoid  redundant  computation,  the  3x3  block  averages  (if  a  7x7  window 
is  chosen)  for  the  entire  image  are  computed  and  stored  in  a  bitmap  equal  to  the  size  of 
the  original  bitmap.  Hence,  when  the  average  color  value  of  a  3x3  block  is  needed,  it  is 
simply  looked  up  within  this  bitmap  rather  than  being  computed  on  the  fly.  This  also 
allows  the  input  image  to  be  overwritten  with  the  output  image  without  effecting  block 
averages  during  processing. 

In  addition  to  least  square  deviations  in  RGB  color  space,  color  quantization  is  another 
technique  employed  here  which  effects  the  algorithm  s  propensity  to  smooth  similar 
pixels  that  are  adjacent  to  one  another.  Color  values  are  quantized  on  values  equivalent 
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to  powers  of  2.  This  allows  quantized  values  to  be  derived  via  a  simple  addition  and  shift 
operation.  An  addition  of  one  half  the  quantization  value  is  applied  prior  to  the  shift  to 
guarantee  that  colors  are  quantized  to  the  nearest  quantized  value.  Additionally,  since 
MMX  registers  and  instructions  are  utilized,  this  operation  can  be  performed  in  parallel 
for  the  red,  green,  and  blue  components  of  a  color.  It  should  also  be  noted  that  after 
quantization  the  color  space  is  reduced  (e.g.  values  ranging  from  0-255  now  range  from 
0-63  if  a  quantization  value  of  4  is  chosen).  This  is  important  to  consider  when  choosing 
a  value  for  zsm,  a  threshold  constant  that  is  compared  to  the  minimum  and  maximum 
color  contrasts  computed.  This  constant  effectively  discriminates  pixels  into  one  of  three 
categories:  noise  pixels  (min.  and  max.  contrasts  are  both  greater  than  zsm),  edge  pixels 
(min.  contrast  is  less  than  zsm  and  max.  contrast  is  greater  than  zjm),  and  interior  pixels 
(min.  and  max.  contrasts  are  both  less  than  %sm).  As  shown  in  figure  3,  the  smoothing 
operation  depends  on  the  specific  cases  encountered. 


flag  as  a  non-interior  pixel, 
and  smooth  next  pixel 

Figure  3.  Smoothing  Algorithm. 
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3.  SEGMENT  INITIALIZATION  AND  REGION  GROWING 

Initialize  a  set  of  segments  for  the  image  by  performing  efficient  flood  fill  operations 
seeded  at  interior  points  (i.e.  not  flagged  as  non-interior,  see  figure  3).  The  fill  algorithm 
used  is  described  in  [9]  and  is  efficient  in  time  and  space  requirements  since  it  guarantees 
that  each  pixel  within  a  formed  segment  is  only  evaluated  a  single  time  during  the  flood 
fill.  The  decision  as  to  whether  a  given  adjacent  (4-connected)  pixel  should  be  filled 
during  the  flood  operation  is  also  based  on  its  closeness  in  color  to  the  original  seed  pixel 
for  the  segment.  This  similarity  (or  contrast)  is  again  determined  through  quantization  of 
color  values  and  computation  of  the  least  square  difference  between  the  two  colors  in  the 
RGB  color  space.  As  pixels  are  merged  in  segments  they  are  encoded  with  an  associated 
unique  segment  identifier.  After  the  flood  fill  operation,  most  pixels  within  the  image 
now  belong  to  particular  segments.  Those  pixels  which  have  not  been  merged  with  any 
segment  (i.e.  are  not  encoded  with  a  segment  identifier)  are  merged  through  a  process  of 
region  growing  described  below. 

In  order  to  merge  the  remaining  pixels  currently  unbound  to  a  particular  segment,  the 
edge  pixels  of  the  existing  set  of  segments  are  propagated  outward  or  grown.  Then,  as 
unbound  pixels  are  encountered,  they  are  merged  with  the  closest  segment  of  most 
similar  color.  This  specific  condition  is  calculated  using  a  least  squares  difference  of  the 
red,  green,  and  blue  color  components  as  well  as  a  distance  proportional  to  the  distance 
from  the  original  propagating  edge  pixel.  This  Euclidean  distance  is  quickly 
approximated  by  computing  the  3,4  distance  transform  (DT)  during  propagation  (see 
figure  4).  Propagation  is  terminated  in  a  particular  direction  when  color  values  are  no 
longer  being  replaced  in  that  direction.  Clearly,  this  processing  step  also  creates 
additional  memory  requirements  by  requiring  memory  for  the  storage  of  DT  values 
within  the  bitmap  and  for  a  propagation  stack  used  to  keep  track  of  color  and  DT  values 
currently  propagating. 


3-7 


Figure  4.  Edge  Point  Propagation.  Shown  is  a  smple  example  showing  the  propagation  of  a  color  value  from  skeletal  point  p 
where  the  only  color  value  replaced  is  at  point  ao.  Thus,  the  color  vaiue  and  DT  value  at  ao  propagates  to  bi,  bo,  and  bz  where  the 
only  value  replaced  is  at  point  bi.  Thus,  the  process  continues  again  when  the  color  value  and  DT  value  at  co  is  replaced. 


4.  TEST  APPLICATION 

The  described  segmentation  procedure  was  implemented  for  Windows  95  and  Windows 
NT  and  allows  easy  evaluation  of  results  and  experimentation  with  various  parameters  of 
the  approach.  Multiple  images  may  be  opened  adjacent  to  each  other  and  processed  with 
varied  parameter  settings.  Figure  5  shows  one  screen  shot  where  the  user  may  adjust 


these  parameters. 


3-8 


RTSey  RTSeyt 


Figure  6.  Test  Application  Screen  Shot.  The  top  window  displays  the  original  image  while  the  lower  two  windows 
segmentation  results  when  different  parameter  choices  are  made. 
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5.  PERFORMANCE  RESULTS 

Performance  of  the  methods  presented  here  was  evaluated  by  measuring  the  time 
requirements  of  each  processing  step.  These  timings  were  precisely  obtained  by  using 


the  profiling  registers  built  into  Intel  Pentium  family  processors. 

Timing  results  for 

smoothing  and  region  growing 

are  listed  in  CPU  cycle  counts. 

Image 

Image  size  in 
pixels  (WxH) 

Smoothing 
(2  processors) 

Flood  fill  and 
region  growing _ 

Segmentation 
method  of  [8] 

Lena 
Peppers 
Video  1 
Video  2 

512x512 

512  x 512 

320  x  240 

320  x  240 

28,629,858 
28,934,578 
7.201,471 
7.106.629 _ 

86.044.047 

75,440.006 

24.337,244 

27.121.800 

»  5  minutes 
»  5  minutes 
»  1  minute 
»  1  minute 

Table  1.  Windows  NT  Performance  results  on  a  workstation  with  dual  Pentium  II  300Mhz  processors. 
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6.  CONCLUSIONS 

In  this  paper  a  tool-methodology,  for  adaptive  segmentation  of  color  images,  or  a 
sequence  of  images  in  real-time  was  presented.  This  effort  was  supported  by  an  AFRL- 
Rome  Lab  research  grant  to  develop  a  tool  for  extraction  knowledge  from  segmented 
images  with  application  to  ATR  in  video  images.  The  results  are  promising  for  real-time 
segmentation  and  near  real-time  segmentation  for  video  with  large  size  images. 
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Abstract 

We  report  here  on  our  recent  experiments,  and  results  on  automatic  classification  of  free 
text  documents  into  a  given,  large  number  of  categories.  Text  classification  has 
applications  in  information  filtering  and  routing,  automatic  indexing,  retrieval,  etc.  We 
use  a  latent  semantic  indexing-based  feature  extractor  and  integrate  neural  net  learning 
into  the  method.  We  then  compare  the  performance  with  our  past  experiments  of 
classification  into  a  smaller  number  of  categories.  We  also  conducted  experiments  to 
assess  the  impact  of  the  training  set  size  in  this  context. 

The  feature  extractors  are  based  on  the  “latent  semantics”  of  a  reference  library. 
Intuitively,  the  technique  of  latent  semantic  indexing  (LSI)  Peerwester,  et  al.,  90] 
projects  any  document  into  a  dimensionally  reduced  space  of  concepts  from  the  reference 
library.  Neural  nets  are  used  to  incorporate  a  learning  component,  as  well  as  to  fuse 
information  from  different  combinations.  Metrics  such  as  micro  and  macro  averaged 
precision,  recall  and  correctness  are  used  to  compare  the  performance  of  the  different 
feature  extractors  and  effectiveness  of  their  fusion. 

The  preliminary  results  indicated  that  the  problem  of  classification  into  a  large  number  of 
categories  is  substantially  more  complex.  So,  we  focused  exclusively  on  this  issue,  and 
evaluated  the  impact  of  training  set  size  on  performance.  This  is  an  important  and  new 
direction  of  experimentation,  which  was  indicated  by  earlier  experiments.  Allowing  for 
the  inherent  complexity  of  the  problem,  neural  net  training  seemed  to  have  added  only  a 
little  value  to  straight  LSI-based  results.  A  surprising  observation  was  that  even 
substantial  variation  in  the  training  set  size  had  little  impact  on  the  results.  We  discuss 
possible  explanations  for  this  surprising  behavior. 
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1.  Introduction 

Information  retrieval  and  classification  refer  to  automated  retrieval  in  response  to  a  query  and  automated 
classification  into  various  categories  of  what  may  be  thought  of  as  unstructured  information.  This  endeavor, 
therefore,  is  to  be  contrasted  specifically  with  database  retrieval,  where  the  data  are  structured .  In  this  age  of 
information  explosion,  information  retrieval  and  classification  have  become  more  critical  research  areas  than  ever. 
In  this  work,  our  focus  is  on  automated  classification  of  unstructured  text.  Such  automated  classification  has  many 
applications,  such  as  information  filtering  based  on  user  profiles,  personalized  information  delivery,  etc.  For  the 
problem  of  information  retrieval  itself,  classification  of  information  can  help  in  automatic  indexing,  as  well  as  in 
other  ways.  Other  related  applications  include  classification  and  routing  electronic  message  traffic  (e.g.,  in 
military  aircraft  operations),  e-mail  filtering  (very  useful  when  a  department  receives  thousands  of  e-mail  messages 
a  day),  etc. 

Thus,  our  initial  goal  in  this  work  was  to  identify  some  new  “feature  extractors”  that  extract  important  features 
from  text,  and  also  to  evaluate  the  effectiveness  of  the  individual  feature  extractors,  and  that  of  information  fusion 
from  multiple  such  feature  extractors,  in  automatically  classifying  free  text  documents  into  a  given  number  of 
categories.  This  was  consistent  with  the  prior  work  that  we  had  done  at  the  Air  Force  Rome  Laboratory.  During 
that  work,  we  had  also  identified  a  couple  of  other  important  issues.  First,  scaling  the  approach  to  a  larger 
number  of  categories  has  been  an  important  concern.  Since  the  Reuters  data  lends  itself  to  experiments  on 
classification  into  a  larger  number  of  categories,  we  focused  on  this  issue.  Neural  net  learning  was  incorporated 
into  the  approach  so  that  information  derived  by  a  feature  extractor  from  training  documents,  along  with  the 
correct  classification,  are  used  to  train  the  neural  network.  With  neural  net  training  in  place,  we  returned  to  an 
issue  that  was  indicated  by  our  previous  experimentation,  namely,  evaluating  the  impact  of  training  set  size  on 
performance. 
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We  start  with  the  standard  feature,  namely,  term  frequency.  We  then  project  a  document  vector  into  the  space  of 
semantic  concepts  from  a  reference  library  using  Latent  Semantic  Indexing  (LSI)  [Deerwester,  et  al.,  90].  A  term- 
document  (or  a  phrase-document)  reference  matrix  represents  the  frequencies  of  terms  (or  phrases,  respectively) 
from  a  collection  of  documents  from  the  reference  library.  In  LSI,  this  large  and  sparse  term-document  matrix  is 
reduced  into  three  smaller  matrices,  one  of  which  is  simply  a  diagonal  matrix,  by  singular  value  decomposition 
(SVD).  The  diagonal  matrix  yields  the  singular  values  from  which  the  dominant  ones  are  selected.  Thus,  instead 
of  representing  documents  by  thousands  of  possible  terms  or  phrases,  LSI  allows  for  a  document  to  be  represented 
by  a  substantially  smaller  number  of factors  that  intuitively  capture  the  significant  “latent  semantics.” 

Thus,  a  reference  matrix  is  the  term-document  matrix  of  a  reference  libraiy  /  collection  of  documents.  A  reference 
library  is  the  collection  of  documents  that  adequately  represents  all  concepts  of  interest.  The  SVD  computations 
are  performed  on  such  a  reference  library  only  once,  and  the  linear  transformations  mentioned  above  essentially 
project  any  new  document  into  the  concept  space  represented  by  the  reference  library.  One  of  the  parameters  we 
experimented  with  in  our  past  work  is  the  size  of  the  reference  library. 

Section  2  describes  background  material  and  the  general  approach.  It  also  discusses  the  metrics  we  use.  Section  3 
deals  with  the  experimental  set  up.  Section  4  summarizes  and  discusses  the  results.  Section  5  concludes  the  paper 
by  putting  this  work  in  the  context  of  other  related  work.  Appendices  A  through  D  include  various  tables  that 
show  details  of  performance  for  each  experiment  and  other  details. 

2.  Background 

Our  hypothesis  in  this  work  has  been  that  a  neural  network,  as  in  GRAIL  [Uberbacher  and  Mural,  91]  [Xu,  et  al., 
94],  would  incorporate  a  learning  component  into  a  classification  method,  besides  possibly  integrating  information 
from  different  features  in  a  systematic  way,  thereby  improving  performance  over  previous  approaches.  To  this  end, 
our  specific  goal  in  this  work  has  been  to  investigate  the  problem  of  classifying  text  documents  into  a  large  number 
of  categories,  measure  the  performance,  and  compare  /contrast  it  against  that  involving  a  smaller  number  of 
categories.  In  order  to  make  training  the  neural  net  practical,  the  primary  issue  is  containment  of  the 
dimensionality  of  the  pattern  recognition  problem,  and  Latent  Semantic  Indexing  (LSI)  [Deerwester,  et  al.,  90]  is 
used  for  this  purpose.  (Thus,  there  are  two  ways  to  view'  the  approach:  either  as  a  neural  net  method,  made 
practical  by  LSI,  or  as  an  LSI-based  approach  augmented  with  neural  net  learning.)  We  first  present  a  quick 
overview  of  how  LSI  is  used  in  this  work  and  then  briefly  describe  the  metrics  used  to  measure  performance, 
adopted  mostly  from  [Dasigi.  97]. 

2.1.  Dimensionality  Reduction  with  LSI 
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The  developers  of  LSI  indicate  that  a  query  may  be  viewed  as  a  pseudo-document  and  may  be  represented  as  a 
vector  of  a  chosen  number  of  factors,  so  it  may  be  placed  in  the  same  vector  space  as  regular  document  vectors 
Peerwester,  et  al.,  90],  This  technique  is  used  in  here  to  reduce  the  dimensionality  of  a  regular  document,  so  it 
can  be  input  to  a  neural  network.  This  is  done  as  follows.  First,  as  pointed  out  in  the  introduction,  a  reference 
term-document  sparse  matrix  X  is  derived  from  the  library  of  documents  that  are  of  interest.  This  matrix  is  split 
into  three  matrices  by  Singular  Value  Decomposition  (SVD)  [Forsythe,  et  al.,  77],  so  that 


X  =  T.S.D' 


Here,  X  is  a  txd  matrix,  where  t  is  the  numbers  of  distinct  terms  (word  roots)  and  d  is  the  number  of  documents  in 
the  reference  collection.  The  order  of  T  is  txk,  that  of  S,  which  is  a  diagonal  matrix,  is  kxk,  and  that  of  D  is  dxk, 
where  k  is  the  chosen  number  of  factors.  (LSI  research  indicates  that  a  choice  of  k  near  100  is  good.)  Now,  the 
pseudo-document  vector  Dq corresponding  to  a  lxt  query  vector  Q  may  be  derived  simply  as: 

Dq  =  Q.T.S-‘ 

In  this  work,  we  use  this  same  idea  to  squash  any  lxt  document  vector  into  a  lxk  vector  that  serves  as  input  to  the 
neural  network.  The  dimensionality  reduction  is  substantial:  t  is  generally  a  few  thousand  to  a  few  tens  of 
thousands,  and  k  is  near  one  hundred.  The  only  care  that  must  be  exercised  is  to  make  sure  that  the  reference 
term-document  matrix  that  is  used  as  the  starting  point  is  one  that  “ adequately  ”  represents  all  concepts  of 
interest.  (We  return  to  address  the  adequacy  issue  in  Section  4  on  Experiments  and  Results.)  This  requirement  is 
no  more  stringent  than  would  be  required  in  the  standard  LSI  approach. 

2.2.  Performance  Metrics 

Precision  and  recall  are  very  commonly  used  as  measures  of  performance  in  information  retrieval.  Precision  is 
defined  as  the  fraction  of  retrieved  items  that  are  actually  relevant  (the  rest  are  false  positives),  and  recall  is 
defined  as  the  fraction  of  relevant  items  that  are  actually  retrieved  (the  rest  are  false  negatives).  Thus,  precision  is 
inversely  related  to  false  positives  and  recall  is  inversely  related  to  false  negatives  during  retrieval. 

Lewis  is  perhaps  the  first  to  define  these  metrics  in  the  context  of  information  classification  [Lewis,  92],  In 
classification,  there  are  multiple  classes,  zero  or  more  of  which  are  to  be  assigned  to  each  document.  If  each  class 
were  viewed  as  a  retrieval  query,  the  definitions  fall  into  place.  That  is,  for  any  class,  precision  is  the  fraction  of 
items  assigned  to  the  class  that  actually  belong  to  the  class,  and  recall  is  the  fraction  of  items  that  belong  to  the 
class  that  are  actually  assigned  to  the  class.  We  further  define  correctness  of  classification  as  the  fraction  of  all 
documents  that  are  classified  correctly  with  respect  to  the  class,  namely,  the  ratio  of  the  sum  of  the  numbers  of  true 
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positives  and  true  negatives  to  the  total  number  of  documents.  That  is,  for  the  i-th  class,  if  tp„  fp„  tn,  and  fn*  stand 
respectively  for  the  numbers  of  true  positives,  false  positives,  true  negative  and  false  negatives,  we  have: 

precision!  =  tpi  /  (tp,+fp,) 

recall,  =  tpi  /  (tpj+fhi) 

correctness,  =  (tp.+tni)  /  (tp,+tn,+fp,+fh;) 

The  picture  gets  a  little  more  complex.  With  these  metrics  distributed  over  all  the  classes,  to  get  global 
performance  measures,  some  kind  of  averaging  is  needed.  There  are  two  ways  to  do  the  averaging.  In  macro- 
averaging,  the  individual  measures  for  each  class  are  taken  and  an  overall  average  is  computed  simply  by  adding 
them  and  dividing  the  grand  total  by  the  number  of  classes.  In  micro-averaging ,  the  grand  total  aggregates  of  the 
individual  numbers  for  true/false  positives  and  true/false  negatives  are  computed  and  the  appropriate  ratios  give 
the  metrics.  Thus,  assuming  below  that  i  ranges  over  all  classes  in  the  summation,  with  the  scope  of  E  being  the 
single  term  or  parenthesized  expression  following  it.  and  N  is  the  number  of  classes,  we  have: 


macro-precision 

macro-recall 

macro-correctness 


E,  precision,  /  N 
E,  recall,  /  N 
Ej  correctness,  /  N 


micro-precision  =  E;  tpi  /  E,  (tp,+fp,) 

micro-recall  =  E,  tp,  /  E,  (tp,+fn,) 


micro-correctness  =  E,  (tp,+tn,)  /  E,  (tp,+tn,+fp,+fn,) 


A  simple  analysis  reveals  that  macro-averaging  and  micro-averaging  always  result  in  the  same  value  for 
correctness.  Another  point  to  note  is  that  the  correctness  measure  can  increase  just  by  increasing  true  negatives, 
without  directly  affecting  the  other  measures.  Since  generally  the  number  of  true  negatives  is  very  high,  a  high 
correctness  measure  can  give  a  false  /  misleading  sense  of  good  performance. 

Generally,  in  any  retrieval  /  classification  experiment,  a  single  parameter  is  varied  to  get  a  precision-recall  pair  at 
each  parameter  value.  Together,  they  form  a  precision-recall  curve.  Generally,  in  such  experiments,  when  the 
parameter  is  changed  so  as  to  increase  precision,  the  recall  figure  decreases  and  vice  versa.  In  the  extreme  case, 
when  the  parameter  is  such  that  there  are  no  false  positives  (100%  precision),  very  likely  we  recall  very  little  by 
being  conservative.  On  the  other  hand,  if  the  parameter  is  such  that  there  are  no  false  negatives  (100%  recall),  we 
are  so  permissive  that  we  retrieve  everything,  decreasing  precision  substantially. 
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Due  to  this  inverse  relationship  between  precision  and  recall,  it  is  hard  to  compare  the  performance  of  different 
approaches  by  looking  at  isolated  figures.  The  entire  precision-recall  curves  need  to  be  compared.  Alternatively,  a 
single  measure  called  the  break-even  point  is  sometimes  used  for  comparison.  The  break-even  point  is  the  point  on 
the  precision  recall  curve  that  has  the  same  value  for  precision  and  recall.  A  higher  break-even  precision  (recall) 
indicates  better  performance.  The  break-even  point  may  be  linearly  interpolated  from  the  two  points  closest  to  it 
on  either  side  on  the  precision-recall  curve.  If  the  two  points  on  either  side  of  the  break  even  point  are  <pi,  rj>  and 
<P2,  r2>  such  that  pi>ri  and  p2<r2,  the  break-even  point  <b,  b>  is  given  by: 


b  =  (r2*pi  -  ri*p2)  /  (r2-r,  +  prp2) 


3.  Experimental  Set  Up 

In  order  to  train  the  neural  network,  we  turn  to  a  collection  of  data  made  available  by  the  Reuters  news  agency,  and 
prepared  by  Dr.  David  Lewis2.  The  Reuters  data  are  described  in  Pasigi  et  al.,  97]  and  [Lewis,  92],  The 
advantage  of  this  collection  is  that  the  Reuters  stories  are  already  manually  classified,  and  include  a  sufficiently 
high  number  of  stories  per  category.  The  Reuters  stories  have  anywhere  from  zero  to  five  categories  assigned  to 
each  of  them.  Since  there  are  about  135  categories  for  the  Reuters  data,  we  originally  combined  several  of  them 
into  single  “higher  level”  categories,  as  suggested  by  the  data  developers;  our  past  results  using  these  higher  level 
categories  were  impressive,  as  reported  in  Pasigi,  97],  The  categories  used  now  appear  in  an  Appendix  D;  the 
higher  level  ones  that  we  had  used  appear  below: 

1.  money  /  foreign  exchange 

2.  shipping 

3.  interest  rates 

4.  economic  indicators 

5.  currency 

6.  corporate 

7.  commodities 

8.  energy 


2  The  Reuters-22173  text  categorization  collection  is  copyrighted  by  Reuters,  Ltd.,  and  is  distributed  freely  for 
research  purposes  only.  Copyright  for  additional  annotations  and  a  number  of  auxiliary  files  resides  with  David 
D.  Lewis  and  the  Information  Retrieval  Laboratory  at  University  of  Massachusetts.  The  authors  thank  the 
providers  of  the  Reuters-22173  collection  for  making  it  available.  It  can  be  obtained  at  the  URLflp://ciir- 
ftp.  cs.  umass.  edu/pub/reutersl . 
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The  size  of  the  Reuters  corpus  is  about  25  MB,  representing  22,173  Reuters  news  wire  stories.  The  stories  are  pre¬ 
classified  and  pre-analyzed.  Thus,  in  addition  to  the  25  MB  of  raw  text,  many  additional  files  are  provided  that 
contain  information  such  as  what  categories,  if  any,  have  been  assigned  to  each  document,  what  words  are  present 
in  each  document  (story)  with  what  frequency,  what  multi-word  noun-phrases  occur  in  each  document  with  what 
frequency,  etc.  Many  details  about  the  collection  can  be  found  in  [Lewis,  92],  and  also  in  the  README  file 
associated  with  the  collection.  Details  of  distribution  of  the  documents  among  the  various  categories  may  be  found 
in  [Dasigi,  et  al.,  97], 

The  collection  contains  22,791  different  words  and  29,496  different  noun-phrases  indexed  to  the  documents. 
These  numbers  are  rather  high,  and  part  of  the  reason  is  “words”  corresponding  to  the  different  numeric  values 
that  occur  in  the  documents,  special  symbols,  etc.  have  all  been  treated  as  genuine  words.  It  is  up  to  the  system 
that  processes  this  information  to  use  them  appropriately.  (In  our  work,  we  do  not  do  anything  special  in  this 
regard.)  The  documents  come  with  pre-assigned  classification,  with  about  135  topic  categories  and  another  539 
other  kinds  of  categories. 

As  mentioned  before,  the  main  feature  extractor  used  in  this  work  is  based  on  LSI  applied  to  word  frequencies. 
The  choice  of  the  reference  library  can  be  important  in  LSI,  because  documents  are  represented  in  the  space 
defined  exclusively  by  the  concepts  from  the  reference  library'.  The  reference  library  that  we  used  comprises  the 
first  3,000  documents. 

4.  Experiments  and  Results 

We  start  off  by  pointing  out  the  precision  and  recall  values  when  straight  LSI-based  retrieval  is  adopted  to  perform 
classification  into  the  eight  higher  level  categories  -  an  easier  problem.  This  is  done  by  first  using  LSI  to  retrieve 
the  document  from  the  reference  library  that  best  matches  the  document  to  be  classified.  In  other  words,  the 
document  to  be  classified  is  used  as  if  it  were  a  query,  as  explained  in  [Deerwestcr,  et  al.,  90],  Then  the  categories 
assigned  to  the  best-matching  document  from  the  reference  library'  are  assigned  to  the  “query  document”, 
completing  the  classification  task.  No  learning  is  incorporated  into  this  method.  In  this  experiment,  there  are  no 
parameters  to  vary  as  in  the  other  experiments,  and  so  there  is  a  single  set  of  macro-  and  micro-  averaged  precision 
and  recall  figures,  and  correctness.  The  results  are  as  follows: 

Macro-Precision  Macro-Recall  Micro-Precision  Micro-Recall  Correctness 

9.55%  5.60%  20.86%  13.56%  89.76% 

It  may  be  noted  that  although  the  correctness  figure  is  good,  the  precision-recall  figures  are  not  very  high.  One 
reason  for  this  behavior  might  be  that  the  number  of  true  negatives  is  perhaps  very  dominating,  pushing  the 
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correctness  percentage  very  high.  Since  true  negatives  have  no  bearing  on  precision  and  recall,  those  figures  are 
small.  In  particular,  it  may  be  noted  that  in  either  kind  of  averaging,  both  the  precision  and  recall  figures  are 
rather  low,  a  behavior  that  may  be  contrasted  with  the  results  from  the  remaining  experiments  that  (i)  incorporate 
learning  and  (ii)  attempt  classification  into  the  fiill  contingent  of  135  categories,  rather  than  into  just  eight. 

The  number  of  output  units  in  all  neural  net  experiments  is  135,  one  for  each  class.  Coincidentally,  the  maximum 
number  of  factors  that  could  be  used  from  the  singular  value  decomposition  (SVD)  computations  was  also  1353. 
The  neural  nets  therefore  also  have  135  input  units.  The  number  of  hidden  units  used  is  9  in  most  of  the 
experiments.  The  specific  architecture,  namely  the  number  of  hidden  layers  (always  one  in  our  experiments)  and 
the  number  of  hidden  units,  is  chosen  after  some  experimentation,  based  on  performance  after  initial  training. 
(Several  meaningful  choices  have  been  experimented  with  for  this  purpose.  Specifically,  we  also  report  on  some 
experimentation  with  a  slightly  larger  number  of  hidden  units,  namely,  15,  as  discussed  later.)  All  the  networks 
use  the  hyperbolic  tangent  for  transfer  function  and  the  delta  rule  for  learning. 

Of  the  22,173  documents,  the  first  4,000  documents  are  initially  used  for  training  the  neural  network,  and  the 
remaining  18,173  are  used  for  testing.  The  neural  network  is  trained  for  16,000  iterations  at  a  time  (each  training 
document  presented  four  times),  all  the  way  up  to  about  192,000  iterations.  After  each  step  of  16,000  iterations, 
the  network  is  tested  with  all  the  training  documents  and  then  with  all  the  test  documents,  separately,  and  the 
outputs  saved  in  files.  These  raw  output  files  are  interpreted  with  the  threshold  value  (that  distinguishes  the  binary 
decision)  as  the  parameter  and  macro-  and  micro-averaged  precision  and  recall,  as  well  as  correctness,  are 
calculated  at  each  parameter  value.  Break-even  points  are  also  calculated  for  the  macro-averaged  precision-recall 
curve  as  well  as  the  micro-averaged  precision-recall  curve.  The  performance  measures  presented  here  are  those  for 
the  test  data  set  that  correspond  to  the  same  point  of  training  that  results  in  the  best  break-even  points  in  the 
training  data. 

Then  we  repeat  the  experiments  by  increasing  the  training  set  size  by  the  next  4,000  documents.  In  other  words, 
the  next  4,000  documents  are  moved  from  the  test  set  to  the  training  set.  Thus,  when  8,000  documents  are  used  for 
training,  only  14,  173  documents  are  used  for  testing.  The  other  experiments  use  training  set  sizes  of  12,000, 
16,000,  and  20,000,  with  test  set  sizes  of  10,173,  6,173,  and  2,173,  respectively.  These  experiments  are  called 
experiments  1  through  5,  with  experiment  1  using  4,000  training  documents  and  experiment  5  using  20,000 
training  documents. 

The  relevant  results  are  distributed  over  several  tables,  which  are  included  with  the  report.  Summary  tables  with 
breakeven  points  have  been  created  for  each  experiment  and  reported  in  Appendix  A.  Whenever  additional 
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information  is  required  (e.g.,  correctness,  precision-recall  measures  for  other  threshold  points),  it  is  included  as 
appropriate.  A  preliminaiy  analysis  of  the  raw  Reuters-22173  data  shows  that  of  the  22,173  documents  1 1664 
(53%)  documents  have  topics  assigned  to  at  least  one  category  and  10509  documents  were  not  assigned  at  all.  The 
point  to  note  here  is  that  a  significant  portion  of  the  data  (47%)  does  not  belong  to  any  of  the  135  categories.  These 
results  can  be  found  in  Appendix  C. 

As  pointed  out  in  the  section  on  Performance  Metrics,  the  raw  data  tables  for  macro/micro  precision-recall  results 
(Appendix  B)  show  that  macro-averaging  and  micro-averaging  result  in  the  same  correctness  values  for  all 
threshold  values.  For  negative  threshold  values,  correctness  has  small  values  (about  1%),  recall  has  large  values 
(about  100%)  and  precision  has  small  (about  1%).  For  those  thresholds  values  there  are  no  false  negatives,  which 
implies  that  everything  was  retrieved,  decreasing  precision  substantially.  However,  as  the  threshold  value  increases 
by  .01,  correctness  improved,  precision  increases  (up  to  100%)  while  recall  decreases  and  a  breakeven  point  is 
noted  at  sometime.  From  macro/micro  precision-recall  results  one  could  observe  that  there  is  clearly  an  inverse 
relationship  between  precision  and  recall. 

For  clarity,  simplicity  and  comparison,  the  breakeven  points  for  all  the  experiments  were  compiled  into  ten  tables 
(see  Appendix  A).  Each  table  contains  an  experiment  number,  the  number  of  documents  (training/testing),  the 
number  of  learning  iterations  and  the  micro/macro  measures  (breakeven,  threshold  and  correctness  points).  The 
results  show  that  the  macro-average  breakeven  points  are  in  the  range  6. 14  - 16.44  for  training  results  and  4.71  - 
8.59  for  testing.  The  micro-average  breakeven  points  range  from  28.12  to  36.82  for  training  and  31.58  to  33.53  for 
testing.  A  point  to  note  is  that  a  higher  breakeven  point  indicates  an  improvement  in  both  precision  and  recall. 

The  best  breakeven  points  for  testing  with  training  data  came  from  experiment  1. 


Experiment  1  Training  -  4000  Testing  -  18173  Performance  when  tested  with  training  data 


MACRO 

Threshold 

MICRO 

Threshold 

Learning 

Breakeven 

(Occurred 

Correctness 

Breakeven 

(Occurred 

Correctness 

Iterations 

(%) 

Between) 

(%> 

(%> 

Between) 

(%> 

128000 

16.44 

0.02-0.03 

96.29  -  97.29 

176000 

36.82 

0.13-0.14 

99.52-99.56 

However  the  results  also  show  that  when  in  Experiment  1,  the  network  was  tested  with  18173  test  documents  the 
results  were  not  the  overall  best  of  the  five  experiments.  Better  testing  results  came  from  experiment  2  -5  with 
experiment  5  showing  the  best  macro-average  and  experiment  2  showing  the  best  micro  average.  This  observation 


3  The  exact  choice  of  this  number  is  arbitrary;  a  number  of  about  100  is  recommended  by  the  developers  of  LSI. 
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is  consistent  with  the  expectation  that  a  larger  training  set  can  positively  influence  the  performance  of  the  neural 
network  for  by  the  best  macro-averaged  results  (Experiment  5).  Since  neural  networks  tend  to  be  biased  toward  the 
more  “populous”  categories,  it  is  perhaps  more  important  to  focus  on  the  macro-averaged  performance.  However, 
the  macro-breakeven  points  are  too  small  to  be  sufficiently  impressive.  Interestingly,  the  best  micro-averaged 
values  occur  in  Experiment  2. 

Experiment  5  Training  -  20000  Testing -2173  Performance  with  test  data 


Learning 

Iterations 

MACRO 

Breakeven 

(%) 

Threshold 

(Occurred 

Between) 

Correctness 

(%) 

144000 

8.59 

0.01-0.02 

93.44  -  96.39 

Experiment  2  Training  -  8000  Testing  -  14173  Performance  with  test  data 


Learning 

Iterations 

MICRO 

Breakeven 

(%) 

Threshold 

(Occurred 

Between) 

Correctness 

(%) 

144000 

33.53 

0.13-0.14 

99.26-99.31 

5.  Conclusion 

In  micro-averaging  calculations,  all  categories  used  in  the  classification  are  viewed  equally  important  while  in 
macro-average  calculations  the  categories  with  more  documents  assigned  to  them  dominate.  Since  neural  networks 
can  be  over-trained  or  biased  toward  the  more  populous  categories,  the  macro-average  performance  is  a  better 
measure  of  precision-recall  in  most  categories,  not  just  the  specific  or  populous  ones.  The  results  of  experiment  5 
could  be  significant  from  this  perspective;  however,  the  macro-breakeven  point  values  are  somewhat  small. 

The  overall  percentages  of  correct  result  for  all  breakeven  points  in  the  experiments  were  very  good,  with 
correctness  values  ranges  from  91.60  -  99.70  %.  In  preliminary  analysis,  this  seems  to  demonstrate  that  the  neural 
network  performs  the  task  of  classification  correctly  for  the  known  threshold  point.  However,  these  impressive 
numbers  are  attributable  to  a  large  number  of  true  negatives  (that  is  documents  not  belonging  to  a  category  are 
classified  as  such). 

Extending  the  success  of  the  approach  from  the  original  task  of  classification  into  8  categories  to  the  more  complex 
task  of  classification  into  135  categories  proved  to  be  very  hard.  The  new  problem  is  admittedly  much  harder,  and 
the  neural  network  requires  very  carefully  controlled  training.  Neural  network  literature  indicates  that  the  size  of 
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the  training  set  needs  to  be  much  larger,  too.  After  several  weeks  of  careful  planning  and  experimentation,  the 
performance  was  still  not  entirely  satisfactory.  The  micro  breakeven  point  is  somewhat  better  than  the 
performance  obtained  with  straight  LSI  with  just  8  categories,  although  such  an  achievement  with  the  macro 
breakeven  point  would  have  been  of  better  interest.  A  tentative  conclusion  that  may  be  drawn  at  this  time  is  that 
either  more  experimentation  is  needed  to  integrate  neural  network  training  with  LSI  or  the  neural  network 
approach  reaches  its  limitation  as  the  complexity  of  the  classification  decision  is  increased. 

At  this  point  it  was  decided  to  change  the  architecture  of  the  neural  network  so  as  to  increase  the  number  of  hidden 
units  from  9  to  15.  This  increase  is  more  in  concert  with  the  recommendation  from  NeuralWorks  User  Manual. 
However,  there  was  no  substantial  difference  in  performance.  These  tables  for  Experiments  1  and  5  for  the  revised 
architecture  are  also  included  in  Appendix  A,  at  the  end,  with  appropriate  headers. 

To  compare  our  approach  itself  with  other  similar  methods,  we  believe  our  approach  makes  for  a  synergy  between 
LSI  and  neural  network  learning.  The  dimensionality  reduction  by  LSI  limits  the  size  of  the  neural  network 
making  it  practical,  and  the  neural  network  augments  LSI  with  learning.  Using  a  reference  library  with  a  one  time 
SVD  computation  makes  the  approach  practical.  [Schutze,  et  al.,  95]  also  employ  an  LSI-based  approach  in 
conjunction  with  neural  network  learning.  Our  approach  differs  from  theirs  in  using  a  reference  library  and  in 
employing  multiple  sensors.  In  their  work,  they  seem  to  have  used  the  set  of  all  tested  documents  together  as  the 
reference  library,  obviating  the  need  for  a  hidden  layer  in  the  neural  network.  We  believe  our  approach  is  more 
practical  since  the  SVD  computations  are  made  just  once. 

To  summarize,  in  this  work,  we  shifted  focus  a  little  into  three  areas:  (i)  classification  into  a  large  number  of 
categories,  (ii)  evaluation  of  impact  of  training  set  size,  and  (iii)  impact  of  some  changes  in  the  number  of  hidden 
units  in  the  neural  network.  Overall,  unfortunately,  performance  has  not  scaled  very  well.  One  possible  reason  for 
this  came  into  light  while  the  work  was  in  progress:  Dr.  David  Lewis  announced  that  the  original  Reuters-22173 
data  had  some  anomalies  in  it.  We  decided  to  continue  our  experimentation  with  the  same  data  for  the  reason  that 
the  new  data  that  replaced  the  old  collection  was  not  pre-analyzed  (requiring  substantial  initial  time  commitment) 
and  also  because  we  wanted  to  be  able  to  compare  the  old  and  new  results.  We  made  some  attempts  (including 
personal  communication)  to  find  out  the  nature  of  the  anomalies  in  the  original  Reuters  data,  but  no  information 
was  available. 
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Appendix  A 


MACRO  /  MICRO  BREAKEVEN  RESULTS  FOR  PRECISION  AND  RECALL 


Experiment  1 


Training  Data  Number  of  documents:  4000 


Learning 

Iterations 

MACRO 

Breakeven 

(%) 

Threshold 

(Occurred 

Between) 

Correctness 

(%) 

MICRO 

Breakeven 

(%) 

Threshold 

(Occurred 

Between) 

Correctness 

(%) 

16000 

10.18 

0.04  -  0.05 

96.42  -  97.74 

27.54 

0.07-0.08 

99.27  -  99.48 

32000 

10.99 

0.03  -  0.04 

97.49-98.30 

32.10 

0.14-0.15 

99.44  _  99.52 

48000 

14.22 

0.02-0.03 

95.36-97.22 

34.03 

0.14-0.05 

99.45  -  99.53 

64000 

15.24 

0.02  -  0.03 

95.01  -96.59 

35.56 

0.14-0.15 

99.46  -  99.52 

80000 

13.59 

0.03  -  0.04 

97.22  -  97.95 

36.04 

0.11-0.12 

99.46  -  99.52 

96000 

13.73 

0.03  -  0.04 

97.47  -  98.09 

36.03 

0.11-0.12 

99.48  -  99.52 

112000 

13.41 

0.03  -  0.04 

97.54-98.15 

36.49 

0.12-0.13 

99.50  -  99.55 

128000 

16.44 

0.02  -  0.03 

96.29  -  97.29 

36.56 

0.12-0.13 

99.49  -  99.54 

144000 

14.45 

0.03  -  0.04 

97.52-98.13 

36.55 

0.12-0.13 

99.48  -  99.52 

160000 

14.38 

0.03-0.04 

97.53-98.15 

36.23 

0.12-0.13 

99.51  -99.55 

176000 

16.26 

0.02  -  0.03 

96.19-97.34 

36.82 

0.13-0.14 

99.52  -  99.56 

192000 

16.20 

0.02-0.03 

96.22  -  97.35 

36.65 

0.12-0.13 

99.52-99.56 

Testing  Data  Number  of  documents:  181 73 


MACRO 

Threshold 

MICRO 

Threshold 

Learning 

Breakeven 

(Occurred 

Correctness 

Breakeven 

(Occurred 

Correctness 

Iterations 

(%) 

Between) 

(%> 

(%) 

Between) 

(%> 

16000 

5.00 

0.04  -  0.05 

96.56-97.78 

26.57 

0.07-0.08 

99.10-99.33 

32000 

4.47 

0.03  -  0.04 

97.38  -  98.20 

32.02 

0.13-0.14 

99.25  -  99.33 
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48000 

64000 

80000 

96000 

112000 

128000 

144000 

160000 

176000 

192000 


0.00 

0.04-1.02 

97.93  -  99.49 

0.00 

0.14-1.02 

99.31-99.49 

4.58 

0.04  -  0.05 

97.67  -  98.21 

31.58 

0.13-0.14 

99.24  -  99.30 

0.00 

0.04  - 1.04 

97.81-99.49 

0.00 

0.12-1.04 

99.32  -  99.49 

4.71 

0.03  -  0.04 

97.34  -  97.96 

29.92 

0.11-0.12 

99.28  -  99.33 

4.57 

0.03-0.04 

97.40  -  98.00 

30.44 

0.11-0.12 

99.25-99.31 

4.48 

0.04-0.05 

97.81  -  98.24 

30.32 

0.11-0.12 

99.23  -  99.30 

4.43 

0.03  -  0.04 

97.36  -  97.97 

30.65 

0.12-0.13 

99.28  -  99.33 

4.45 

0.03-0.04 

97.37  -  97.99 

30.31 

0.11-0.12 

99.26  -  99.32 

4.45 

0.04  -  0.05 

97.86  -  98.28 

30.67 

0.12-0.13 

99.26  -  99.32 

4.54 

0.04-0.05 

97.87  -  98.28 

30.64 

0.12-0.13 

99.27  -  99.33 
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Experiment  2 


Training  Data  Number  of  documents :  8000 


Learning 

Iterations 

MACRO 

Breakeven 

(%) 

Threshold 

(Occurred 

Between) 

Correctness 

(%) 

MICRO 

Breakeven 

(%) 

Threshold 

(Occurred 

Between) 

Correctness 

(%) 

16000 

5.69 

0.04  -  0.05 

96.86  -  97.47 

22.54 

0.14-0.15 

99.34-99.43 

32000 

10.80 

0.01-0.02 

91.61-96.21 

24.52 

0.08  -  0.09 

99.34-99.43 

48000 

8.78 

0.02  -  0.03 

95.40-97.39 

23.33 

0.06  -  0.07 

99.17-99.37 

64000 

7.41 

0.02  -  0.03 

96.09  -  97.48 

25.99 

0.10-0.11 

99.27  -  99.39 

80000 

8.86 

0.03-0.04 

97.11  -98.06 

27.89 

0.11-0.12 

99.35-99.45 

96000 

9.56 

0.02-0.03 

95.76  -  97.29 

27.72 

0.14-0.15 

99.33  -99.40 

112000 

9.45 

0.02-0.03 

95.95-97.31 

27.82 

0.13-0.14 

99.37  -  99.46 

128000 

9.45 

0.02-0.03 

95.95-97.31 

27.82 

0.13-0.14 

99.37-99.46 

144000 

10.30 

0.02  -  0.03 

95.62-97.16 

28.04 

0.11-0.12 

99.35-99.45 

160000 

10.17 

0.02-0.03 

95.83  -97.33 

28.12 

0.11-0.12 

99.31  -99.40 

176000 

9.92 

0.02  -  0.03 

95.83  -97.33 

28.01 

0.11-0.12 

99.30-99.39 

192000 

9.94 

0.02-0.03 

95.83  -97.33 

27.91 

0.12-0.13 

99.39-99.48 

Testing  Data 

Number  of  documents:  141 73 

MACRO 

Threshold 

MICRO 

Threshold 

Learning 

Breakeven 

(Occurred 

Correctness 

Breakeven 

(Occurred 

Correctness 

Iterations 

(%) 

Between) 

(%> 

(%) 

Between) 

(%) 

16000 

4.86 

0.05  -  0.06 

97.51  -97.86 

27.22 

0.13-0.14 

99.18-99.29 

32000 

3.78 

0.03  -  0.04 

97.75  -  98.38 

30.25 

0.07-0.08 

99.17-99.30 

48000 

4.22 

0.03-0.04 

97.42-98.31 

28.38 

0.06-0.07 

99.12-99.32 

64000 

4.31 

0.03-0.04 

97.59-98.30 

31.45 

0.10-0.11 

99.24-99.34 

80000 

4.62 

0.03-0.04 

97.21  -98.10 

33.40 

0.10-0.11 

99.24-99.33 

96000 

4.75 

0.03-0.04 

97.38-  98.15 

33.53 

0.13-0.14 

99.26-99.31 

112000 

4.79 

0.03-0.04 

97.43  -98.14 

33.27 

0.11-0.12 

99.25  -99.33 

128000 

4.82 

0.03-0.04 

97.40-98.12 

33.47 

0.12-0.13 

99.28-99.35 

144000 

4.70 

0.03-0.04 

97.25-98.03 

33.28 

0.10-0.11 

99.23  -99.32 
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160000 

4.86 

0.05  -  0.06 

97.51-97.86 

27.22 

0.13-0.14 

99.18-99.29 

176000 

4.83 

0.03-0.04 

97.41-98.16 

33.50 

0.11-0.12 

99.29  -  99.36 

192000 

4.82 

0.03-0.04 

97.41-98.16 

33.49 

0.11-0.12 

99.29  -  99.36 
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Experiment  3 


Training  Data  Number  of  documents:  12000 


Learning 

Iterations 

MACRO 

Breakeven 

(%) 

Threshold 

(Occurred 

Between) 

Correctness 

(%) 

MICRO 

Breakeven 

(%) 

Threshold 

(Occurred 

Between) 

Correctness 

(%) 

16000 

6.45 

0.03-0.04 

95.97-97.34 

22.22 

0.07-0.08 

99.19-99.35 

32000 

7.20 

0.02  -  0.03 

95.91  -97.63 

29.12 

0.18-0.19 

99.33  -99.40 

48000 

5.66 

0.03  -  0.04 

96.84  -  97.84 

29.54 

0.12-0.13 

99.25-99.35 

64000 

6.24 

0.03-0.04 

96.66  -  97.78 

25.93 

0.07-0.08 

99.32-99.45 

80000 

9.36 

0.01-0.02 

91.52-95.87 

29.69 

0.12-0.13 

99.28-99.39 

96000 

6.08 

0.02-0.03 

96.03  -  97.44 

29.77 

0.11-0.12 

99.32  -  99.47 

112000 

7.37 

0.02-0.03 

96.11  -97.51 

29.64 

0.12-0.13 

99.25-99.36 

128000 

7.52 

0.02-0.03 

96.07  -  97.49 

29.68 

0.12-0.13 

99.34-99.47 

144000 

6.87 

0.02-0.03 

95.72  -  97.28 

29.55 

0.11-0.12 

99.29  -  99.43 

160000 

5.17 

0.02  -  0.03 

96.25-97.70 

30.17 

0.13-0.14 

99.30-99.41 

176000 

5.17 

0.02-0.03 

96.21  -97.68 

30.06 

0.13-0.14 

99.31  -99.42 

192000 

5.14 

0.02  -  0.03 

96.19-97.67 

29.97 

0.13-0.14 

99.31  -99.43 

Testing  Data 

Number  of  documents :  101 73 

MACRO 

Threshold 

MICRO 

Threshold 

Learning 

Breakeven 

(Occurred 

Correctness 

Breakeven 

(Occurred 

Correctness 

Iterations 

(%) 

Between) 

(%) 

(%) 

Between) 

(%) 

16000 

3.36 

0.04-0.05 

97.33  -98.27 

23.74 

0.07-0.08 

99.12-99.30 

32000 

3.94 

0.03-0.04 

97.67-98.30 

31.74 

0.17-0.18 

99.25  -  99.30 

48000 

6.22 

0.02-0.03 

95.43-96.93 

31.71 

0.11-0.12 

99.22-99.31 

64000 

4.26 

0.04-0.05 

97.82-98.52 

27.64 

0.06-0.07 

99.01  -  99.27 

80000 

4.53 

0.03-0.04 

97.65  -  98.42 

32.62 

0.12-0.13 

99.26-99.36 

96000 

4.68 

0.03-0.04 

97.52  -  98.24 

32.16 

0.10-0.11 

99.16-99.28 

112000 

4.49 

0.03-0.04 

97.59-98.28 

32.55 

0.12-0.13 

99.21  -99.33 

! 

128000 

5.12 

0.02-0.03 

96.22  -  97.57 

32.36 

0.11-0.12 

99.21  -99.31 

144000 

5.18 

0.03-0.04 

97.38-98.15 

31.83 

0.11-0.12 

99.26  -  99.39 
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160000 

4.30 

0.03-0.04 

97.74  -  98.34 

32.61 

0.13-0.14 

99.27  -  99.37 

176000 

4.32 

0.03  -  0.04 

97.73  -  98.33 

32.54 

0.13-0.14 

99.28  -  99.38 

192000 

4.33 

0.03  -  0.04 

97.72  -  98.33 

32.53 

0.12-0.13 

99.20  -  99.29 

Experiment  4 


Training  Data  Number  of  documents :  16000 


Learning 

Iterations 

MACRO 

Breakeven 

(%) 

Threshold 

(Occurred 

Between) 

Correctness 

(%) 

MICRO 

Breakeven 

(%) 

Threshold 

(Occurred 

Between) 

Correctness 

(%) 

16000 

3.96 

0.03  -  0.04 

97.35-98.06 

25.09 

0.15-0.16 

99.28-99.30 

32000 

5.04 

0.03-0.04 

96.89  -  97.54 

30.39 

0.10-0.11 

99.36  -  99.42 

48000 

4.88 

0.03  -  0.04 

97.39-98.20 

30.57 

0.08-0.09 

99.44  -  99.52 

64000 

5.21 

0.03-0.04 

96.95-97.58 

29.96 

0.13-0.14 

99.34-99.43 

80000 

5.96 

0.02-0.03 

97.60  -  98.28 

31.91 

0.12-0.13 

99.31  -99.38 

96000 

6.14 

0.02  -  0.03 

96.09-97.39 

31.84 

0.12-0.13 

99.30-99.39 

112000 

5.35 

0.02-0.03 

95.98-97.27 

31.96 

0.13-0.14 

99.24-99.30 

128000 

5.69 

0.02-0.03 

95.24-97.36 

31.95 

0.14-0.15 

99.35-99.42 

144000 

5.70 

0.02-0.03 

96.09-97.52 

32.11 

0.12-0.13 

99.32  -  99.39 

160000 

5.69 

0.02-0.03 

95.92-97.38 

32.02 

0.12-0.13 

99.30-99.37 

176000 

5.68 

0.02-0.03 

95.93  -97.39 

31.95 

0.12-0.13 

99.30-99.36 

192000 

5.64 

0.02  -  0.03 

95.95-97.41 

31.94 

0.13-0.14 

99.36  -  99.43 

Testing  Data 

Number  of  documents:  61 73 

MACRO 

Threshold 

MICRO 

Threshold 

Learning 

Breakeven 

(Occurred 

Correctness 

Breakeven 

(Occurred 

Correctness 

Iterations 

(%) 

Between) 

(%) 

(%> 

Between) 

(%) 

16000 

3.77 

0.04-0.05 

98.04  -  98.46 

23.75 

0.13-0.14 

99.16-99.19 

32000 

5.57 

0.03-0.04 

96.97-97.59 

31.72 

0.10-0.11 

99.25-99.31 

48000 

5.00 

0.03-0.04 

97.41  -98.20 

31.47 

0.07-0.08 

99.10-99.26 

64000 

5.38 

0.03-0.04 

97.05-97.65 

31.20 

0.12-0.13 

99.20-99.29 

80000 

5.67 

0.02-0.03 

96.30  -  97.66 

32.94 

0.12-0.13 

99.26-99.34 

96000 

5.28 

0.02-0.03 

96.24-97.47 

32.93 

0.12-0.13 

99.26-99.34 

112000 

5.42 

0.02-0.03 

96.14-97.36 

32.82 

0.13-0.14 

99.25  -99.32 

128000 

5.31 

0.02  -  0.03 

96.11  -97.43 

32.85 

0.13-0.14 

99.25  -99.31 

144000 

5.53 

0.02  -  0.03 

96.25  -  97.59 

33.04 

0.11-0.12 

99.21  -99.27 
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160000 

5.64 

0.02  -  0.03 

96.08  -  97.46 

32.92 

0.12-0.13 

99.26  -  99.33 

176000 

5.66 

0.02  -  0.03 

96.10-97.47 

32.90 

0.12-0.13 

99.25  -  99.32 

192000 

5.66 

0.02-0.03 

96.11-97.49 

32.93 

0.12-0.13 

99.25-99.31 

Experiment  5 


Training  Data  Number  of  documents:  20000 


MACRO 

Threshold 

MICRO 

Threshold 

Learning 

Breakeven 

(Occurred 

Correctness 

Breakeven 

(Occurred 

Correctness 

Iterations 

(%) 

Between) 

(%) 

(%) 

Between) 

(%) 

16000 

4.19 

0.02-0.03 

96.61  -97.16 

25.72 

0.09-0.10 

99.24  -  99.33 

32000 

4.78 

0.03-0.04 

96.97-97.67 

29.97 

0.14-0.15 

99.32-99.42 

48000 

5.38 

0.03-0.04 

96.67  -  97.49 

27.97 

0.09-0.10 

99.18-99.39 

64000 

4.11 

0.02-0.03 

96.10-97.13 

29.39 

0.18-0.19 

99.26-99.34 

80000 

5.31 

0.02  -  0.03 

96.30-97.54 

30.72 

0.13-0.14 

99.29-99.41 

96000 

5.99 

0.02-0.03 

95.77-97.31 

31.01 

0.12-0.13 

99.25-99.42 

112000 

5.79 

0.02-0.03 

96.13-97.40 

30.98 

0.14-0.15 

99.30-99.45 

128000 

6.54 

0.02-0.03 

95.81  -97.31 

31.03 

0.14-0.15 

99.31  -99.48 

144000 

5.50 

0.02-0.03 

96.15-97.60 

31.00 

0.13-0.14 

99.25-99.38 

160000 

8.25 

0.01-0.02 

92.10-95.57 

30.95 

0.15-0.16 

99.29-99.39 

176000 

8.22 

0.01-0.02 

92.06-95.59 

30.96 

0.15-0.16 

99.31  -99.44 

192000 

8.19 

0.01-0.02 

92.02-95.60 

30.82 

0.14-0.15 

99.24  -  99.34 

Testing  Data 

Number  of  documents:  21 

73 

MACRO 

Threshold 

MICRO 

Threshold 

Learning 

Breakeven 

(Occurred 

Correctness 

Breakeven 

(Occurred 

Correctness 

Iterations 

(%) 

Between) 

(%) 

(%) 

Between) 

(%> 

16000 

6.73 

0.01  -  0.02 

95.09  -  96.49 

30.89 

0.08  -  0.09 

99.06-99.21 

32000 

5.19 

0.03-0.04 

97.16-97.79 

31.25 

0.13-0.14 

99.15-99.25 

48000 

5.55 

0.04-0.05 

97.62-98.01 

30.19 

0.09-0.10 

99.11  -  99.31 

64000 

4.90 

0.03-0.04 

97.21  -97.73 

31.06 

ON 

< 

o' 

1 

00 

© 

99.19-99.27 

80000 

5.82 

0.02-0.03 

96.54-97.61 

31.20 

0.12-0.13 

99.11  -99.22 

96000 

6.11 

0.02-0.03 

96.02  -  97.45 

31.16 

0.12-0.13 

99.17-  99.33 

112000 

6.53 

0.02-0.03 

96.36-97.50 

31.91 

0.13-0.14 

99.15-99.23 

128000 

6.51 

0.02  -  0.03 

96.05  -  97.43 

32.44 

0.13-0.14 

99.15-99.24 

144000 

8.59 

0.01-0.02 

93.44-96.39 

32.17 

0.13-0.14 

99.19-99.31 

160000 

6.21 

0.03  -  0.04 

97.21  -97.97 

32.22 

0.14-0.15 

99.14-99.22 
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176000 

6.76 

0.02-0.03 

95.88  -  97.22 

32.23 

0.14-0.15 

99.16-99.25 

192000 

6.70 

0.02  -  0.03 

95.89  -  97.23 

32.12 

0.14-0.15 

99.17-99.28 
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NEURAL  NETWORK  TRAINING  EXPERIMENTS  FOR  LARGE  SCALE 

CLASSIFICATION 


Network  with  135  inputs,  135  outputs  and  15  hidden  nodes. 

Experiment  1 


Training  Data  Number  of  documents:  4000 


Learning 

Iterations 

MACRO 

Breakeven 

(%) 

Threshold 

(Occurred 

Between) 

Correctness 

(%) 

MICRO 

Breakeven 

(%) 

Kffll 

Correctness 

(%) 

16000 

10.81 

0.02  -  0.03 

96.08  -  96.95 

25.09 

0.14-0.15 

99.38-99.43 

32000 

13.64 

0.04  -  0.05 

97.13  -97.94 

31.92 

0.10-0.11 

99.41-99.49 

48000 

18.47 

0.02  -  0.03 

95.75  -  96.99 

34.81 

0.11-0.12 

99.45-99.51 

64000 

19.75 

0.02  -  0.03 

95.99-97.17 

36.26 

0.11-0.12 

99.49  -  99.54 

80000 

18.77 

0.03  -  0.04 

97.41  -  98.03 

38.42 

0.12-0.13 

99.52-99.57 

96000 

19.06 

0.03  -  0.04 

97.29  -  97.94 

38.76 

0.12-0.13 

99.53  -  99.56 

112000 

18.91 

0.03  -  0.04 

97.37  -  98.03 

38.76 

0.11-0.12 

99.48  -  99.54 

128000 

18.85 

0.03  -  0.04 

97.50-98.12 

39.14 

0.12-0.13 

99.53  -99.57 

144000 

19.06 

0.03  -  0.04 

97.43  -  98.07 

38.69 

0.13-0.14 

99.52  -  99.56 

160000 

19.19 

0.03  -  0.04 

97.31  -97.96 

38.37 

0.11-0.12 

99.51  -99.55 

176000 

19.20 

0.03  -  0.04 

97.33  -97.97 

38.76 

0.11-0.12 

99.51  -99.55 

192000 

19.24 

0.03-0.04 

97.34-97.98 

38.86 

0.11-0.12 

99.50  -  99.54 

Testing  Data  Number  o  f  documents:  18173 


Learning 

Iterations 

MACRO 

Breakeven 

(%) 

Threshold 

(Occurred 

Between) 

Correctness 

(%) 

MICRO 

Breakeven 

(%) 

Threshold 

(Occurred 

Between) 

Correctness 

(%) 

4.42 

0.03  -  0.04 

97.07  -  97.59 

27.36 

0.13-0.14 

99.21  -99.28 

5.22 

0.04-0.05 

97.15-97.87 

27.08 

0.10-0.11 

99.23  -  99.31 

6.20 

0.03-0.04 

96.86  -  97.56 

28.79 

0.11-0.12 

99.32-99.37 

64000 

6.47 

0.02-0.03 

95.85  -  97.04 

29.86 

0.10-0.14 

99.21  -99.30 

80000 

5.93 

0.03  -  0.04 

97.25  -  97.89 

30.74 

0.11-0.12 

99.26  -  99.33 

96000 

6.02 

0.03  -  0.04 

97.16-97.80 

29.93 

0.11-0.12 

99.26-99.32 

112000 

5.99 

0.03  -  0.04 

97.20  -  97.86 

30.34 

0.11-0.12 

99.27-99.33 

128000 

5.83 

0.03-0.04 

97.31  -97.34 

30.69 

0.11-0.12 

99.26  -  99.32 

144000 

6.02 

0.03  -  0.04 

97.26  -  97.87 

30.58 

0.12-0.13 

99.26-99.31 

160000 

5.64 

0.03-0.04 

97.15-97.80 

29.44 

0.10-0.11 

99.22  -  99.29 

176000 

5.63 

0.03  -  0.04 

97.17-97.82 

29.76 

0.10-0.11 

99.21  -99.28 

192000 

5.63 

0.03  -  0.04 

97.19-97.83 

29.92 

0.11-0.12 

99.28-99.33 
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Network  with  135  inputs,  135  outputs  and  15  hidden  nodes. 


Experiment  5 


Training  Data 


Number^  of  documents:  20000 


Learning 

Iterations 


MACRO 

Breakeven 


Threshold 

(Occurred 

Between) 


Correctness 


MICRO 

Breakeven 


Threshold 

(Occurred 

Between) 


Correctness 


16000 

32000 

48000 

64000 

80000 

96000 

112000 

128000 

144000 

160000 

176000 

192000 


0.00 

5.95 

3.22 

9.15 
9.35 
6.56 
6.46 

6.16 
6.70 
7.93 
7.90 
7.85 


0.04- 
0.02- 
0.03- 
0.01  - 
0.01- 
0.01- 
0.02- 
0.02 
0.02 
0.01 
0.01 
0.01 


0.99 

0.03 

0.04 

0.02 

0.02 

0.02 

0.03 

0.03 

0.03 

-0.02 

-0.02 

-0,02 


96.60  - 

96.41  - 

98.16- 

94.54  - 

96.47 

94.52 

96.28 

96.56 

96.31 

96.71 

94.39 

94.37 


99.52 
97.61 
98.50 
97.25 

97.52 
96.68 

-97.31 

-97.55 

-98.10 

-  97.68 
-96.71 

-  96.70 


00.00 

30.84 

30.86 

30.24 

31.73 

32.05 

32.15 

32.18 

32.22 

32.22 

32.29 

32.31 


0.17- 

0.12- 

0.12- 

0.15- 

0.13 

0.14 

0.12 

0.14 

0.14 

0.13 

0.13 

0.13 


0.99 

0.13 

0.13 

0.16 

0.14 

0.15 

0.13 

0.15 

-0.15 

-0.14 

-0.14 

-0.14 


99.33  - 

99.32 
99.23 

99.33 

99.29 

99.30 

99.32 
99.29 

99.33 
99.33 
99.33 
99.32 


99.52 

-  99.40 
■  99.34 
-99.38 
-99.38 
-99.39 

-  99.42 
-99.36 

-  99.42 

-  99.42 
-99.41 

-  99.41 


Testing  Data 


Number  of  documents:  2173 


Learning 


Iterations 


MACRO 

Breakeven 


Threshold 

(Occurred 

Between) 


Correctness 


MICRO 

Breakeven 


Threshold 

(Occurred 

Between) 


Correctness 

(%) 


16000 

32000 

48000 

64000 

80000 

96000 

112000 

128000 

144000 

160000 

176000 

192000 


6.73 

5.42 

7.00 

5.64 

7.21 

6.76 

6.66 

6.92 

7.21 

7.09 

7.09 

7.08 


0.05- 

0.02- 

0.02- 

0.02- 

0.02- 

0.02- 

0.03- 

0.02- 

0.02- 

0.02- 

0,02- 

0.02- 


0.06 

0.03 

0.03 

0.03 

0.03 

0.03 

0.04 

0.03 

0.03 

0.03 

-0.03 

0.03 


96.83 

97.06  - 

96,39 

97.32 

96.69 

96.85 

97.43 

96.72 

96.53 

96.91 

96.90 

96.90 


97.23 

98.15 

97.61 

98.13 

-  97.62 

-  97.70 

-  97.99 

-  97.62 

-  97.50 

-  97.74 

-  97.22 

-  97.73 


27.72 

32.04 

31.19 

31.80 

32.57 

32.47 

32.32 

32.75 

32.90 

32.53 

32.55 

32.56 


0.15- 

0.12- 

0.11- 

0.13- 

0.12- 

0.13- 

0.11- 

0.13- 

0.13- 

0.12- 

0.12- 

0.12- 


0.16 

0.13 

0.12 

0.14 

0.13 

0.14 

0.12 

0.14 

0.14 

0.13 

0.13 

0.13 


99.12- 

99.16 

99.17 
99.16 

99.16 

99.17 

99.16 

99.17 
99.20 

99.18 
99.18 
99.18 


-99.23 

-  99.26 

-  99.24 

-  99.21 

-  99.22 

-  99.24 

-  99.25 
-99.23 

-  99.27 

-  99.26 
-99.25 

-  99.25 
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Appendix  B 


Neural  Networks  Training  and  Testing  Data  Results  Example 


SMN32K20.txt  -  Output  for  32000  iteration  for  the  20000  training  set  size. 

Minimum:  -0.074  Maximum:  0.980 

Thres  MacPre  MacRec  MicPre  MicRec  MacCor  MicCor 


-0.066 

0.48 

100.00 

0.48 

100.00 

0.48 

0.48 

-0.056 

0.48 

100.00 

0.48 

100.00 

0.48 

0.48 

-0.046 

0.48 

99.99 

0.48 

99.96 

0.48 

0.48 

-0.036 

0.48 

99.98 

0.48 

99.91 

0.50 

0.50 

-0.026 

0.48 

99.91 

0.48 

99.56 

0.56 

0.56 

-0.016 

0.49 

99.57 

0.47 

98.42 

0.93 

0.93 

-0.006 

0.50 

96.89 

0.47 

95.10 

3.66 

3.66 

0.004 

1.07 

27.75 

2.23 

77.95 

83.48 

83.48 

0.014 

3.45 

11.45 

4.70 

63.89 

93.63 

93.63 

0.024 

5.56 

5.60 

8.72 

55.37 

97.01 

97.01 

0.034 

3.18 

3.23 

13.15 

50.78 

98.16 

98.16 

0.044 

5.15 

2.59 

15.58 

47.99 

98.50 

98.50 

0.054 

5.71 

2.25 

17.15 

45.57 

98.68 

98.68 

0.064 

10.21 

2.04 

18.68 

43.62 

98.82 

98.82 

0.074 

10.67 

1.87 

19.98 

41.98 

98.92 

98.92 

0.084 

15.33 

1.73 

21.16 

40.60 

98.99 

98.99 

0.094 

16.65 

1.65 

22.10 

39.45 

99.04 

99.04 

0.104 

12.78 

1.55 

23.09 

38.23 

99.09 

99.09 

0.114 

15.60 

1.45 

24.41 

36.75 

99.15 

99.15 

0.124 

15.73 

1.32 

26.44 

34.45 

99.23 

99.23 

0.134 

20.20 

1.14 

31.01 

30.74 

99.34 

99.34 

0.144 

17.12 

0.90 

38.26 

25.96 

99.44 

99.44 

0.154 

21.28 

0.75 

47.69 

22.94 

99.51 

99.51 

0.164 

21.61 

0.68 

56.02 

21.56 

99.54 

99.54 

0.174 

19.16 

0.64 

61.38 

20.85 

99.56 

99.56 
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0.184 

0.194 

0.204 

0.214 

0.224 

0.234 

0.244 

0.254 

0.264 

0.274 

0.284 

0.294 

0.304 

0.314 

0.324 

0.334 

0.344 

0.354 

0.364 

0.374 

0.384 

0.394 

0.404 

0.414 

0.424 

0.434 

0.444 

0.454 

0.464 

0.474 

0.484 

0.494 

0.504 

0.514 

0.524 

0.534 


19.12 

18.05 

20.01 

29.14 

23.84 

36.14 
36.51 
73.83 

74.39 
75.07 

75.53 
76.04 

76.62 
77.26 

77.57 

77.85 

78.17 

78.44 
78.99 
79.50 
79.78 
80.04 

80.49 

80.73 
80.95 
81.29 

81.85 

82.15 

82.57 

82.86 
83.09 

83.43 

83.86 
84.07 

84.56 

84.87 


0.63 

0.62 

0.62 

0.62 

0.62 

0.62 

0.62 

0.61 

0.61 

0.61 

0.61 

0.61 

0.61 

0.61 

0.60 

0.60 

0.60 

0.60 

0.60 

0.60 

0.60 

0.60 

0.60 

0.60 

0.60 

0.60 

0.59 

0.59 

0.59 

0.59 

0.59 

0.59 

0.59 

0.59 

0.59 

0.59 


65.34 

67.72 

69.58 

70.61 

71.44 
72.23 
73.01 
73.83 

74.39 
75.07 

75.53 
76.04 

76.62 
77.26 

77.57 

77.85 

78.17 

78.44 
78.99 
79.50 
79.78 
80.04 

80.49 

80.73 
80.95 
81.29 

81.85 

82.15 

82.57 

82.86 
83.09 

83.43 

83.86 
84.07 

84.56 

84.87 


20.59 

20.46 

20.45 

20.38 

20.34 

20.28 

20.24 

20.17 

20.12 

20.07 

20.01 

19.97 

19.91 

19.88 

19.86 

19.80 

19.76 

19.74 

19.70 

19.70 

19.67 

19.65 

19.62 

19.59 

19.55 

19.54 

19.53 

19.49 

19.48 

19.46 

19.44 

19.42 

19.40 

19.37 

19.35 

19.34 


99.57 

99.57 

99.58 

99.58 

99.58 

99.58 

99.58 

99.58 

99.58 

99.59 

99.59 

99.59 

99.59 

99.59 

99.59 

99.59 

99.59 

99.59 

99.59 

99.59 

99.59 

99.59 

99.59 

99.59 

99.59 

99.59 

99.59 

99.59 

99.59 

99.59 

99.60 

99.60 

99.60 

99.60 

99.60 

99.60 


99.57 

99.57 

99.58 

99.58 

99.58 

99.58 

99.58 

99.58 

99.58 

99.59 

99.59 

99.59 

99.59 

99.59 

99.59 

99.59 

99.59 

99.59 

99.59 

99.59 

99.59 

99.59 

99.59 

99.59 

99.59 

99.59 

99.59 

99.59 

99.59 

99.59 

99.60 

99.60 

99.60 

99.60 

99.60 

99.60 
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0.544 

0.554 

0.564 

0.574 

0.584 

0.594 

0.604 

0.614 

0.624 

0.634 

0.644 

0.654 

0.664 

0.674 

0.684 

0.694 

0.704 

0.714 

0.724 

0.734 

0.744 

0.754 

0.764 

0.774 

0.784 

0.794 

0.804 

0.814 

0.824 

0.834 

0.844 

0.854 

0.864 

0.874 

0.884 

0.894 


85.19 

85.37 

85.65 
86.04 

86.28 
86.48 

86.83 

87.35 

87.59 

87.89 

88.39 

88.81 

89.22 

89.65 
90.01 

90.54 

90.83 

91.25 

91.84 
92.07 

92.39 

92.71 

92.73 

92.89 

93.26 
93.47 
93.80 

94.18 

94.51 

94.71 

94.86 
95.21 

95.51 

95.61 
95.91 

95.95 


0.59 

0.59 

0.59 

0.58 

0.58 

0.58 

0.58 

0.58 

0.58 

0.57 

0.56 

0.56 

0.55 

0.55 

0.54 

0.53 

0.52 

0.51 

0.50 

0.49 

0.48 

0.47 

0.46 

0.46 

0.45 

0.45 

0.45 

0.44 

0.44 

0.43 

0.43 

0.43 

0.42 

0.41 

0.40 

0.38 


85.19 

85.37 

85.65 
86.04 

86.28 
86.48 

86.83 

87.35 

87.59 

87.89 

88.39 

88.81 

89.22 

89.65 
90.01 

90.54 

90.83 

91.25 

91.84 
92.07 

92.39 

92.71 

92.73 

92.89 

93.26 
93.47 
93.80 

94.18 

94.51 

94.71 

94.86 
95.21 

95.51 

95.61 
95.91 

95.95 


19.29 

19.26 

19.23 

19.20 

19.11 

19.09 

19.06 

18.95 
18.88 

18.73 

18.54 
18.34 
18.16 

17.95 
17.69 

17.45 
17.15 

16.77 

16.36 
16.07 

15.78 

15.44 

15.19 
15.04 

14.87 

14.72 

14.62 
14.50 

14.38 
14.25 
14.13 
13.98 

13.81 

13.46 
13.06 

12.44 


99.60 

99.60 

99.60 

99.60 

99.60 

99.60 

99.60 

99.60 

99.60 

99.60 

99.60 

99.60 

99.60 

99.60 

99.60 

99.60 

99.59 

99.59 

99.59 

99.59 

99.59 

99.59 

99.59 

99.59 

99.59 

99.59 

99.59 

99.59 

99.59 

99.59 

99.58 

99.58 

99.58 

99.58 

99.58 

99.58 


99.60 

99.60 

99.60 

99.60 

99.60 

99.60 

99.60 

99.60 

99.60 

99.60 

99.60 

99.60 

99.60 

99.60 

99.60 

99.60 

99.59 

99.59 

99.59 

99.59 

99.59 

99.59 

99.59 

99.59 

99.59 

99.59 

99.59 

99.59 

99.59 

99.59 

99.58 

99.58 

99.58 

99.58 

99.58 

99.58 
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0.904 

96.17 

0.35 

96.17 

11.63 

99.57 

99.57 

0.914 

96.09 

0.31 

96.09 

10.26 

99.57 

99.57 

0.924 

96.00 

0.23 

96.00 

7.60 

99.56 

99.56 

0.934 

94.61 

0.11 

94.61 

3.66 

99.54 

99.54 

0.944 

95.22 

0.05 

95.22 

1.54 

99.53 

99.53 

0.954 

91  AO 

0.02 

97.40 

0.58 

99.52 

99.52 

0.964 

92.00 

0.01 

92.00 

0.18 

99.52 

99.52 

0.974 

83.33 

0.00 

83.33 

0.04 

99.52 

99.52 

MACRO  BREAKEVEN  = 

3.22 

occurred  between: 

0.03 

0.04 

MICRO  BREAKEVEN  = 

30.86 

occurred  between: 

0.12 

0.13 
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Appendix  C 

Statistical  Information  about  raw  Reuters-22173  data  before 


creating  input  for  neural  network  experiments 

Number  of  documents  assigned  to  category  1  =  4075 
Number  of  documents  assigned  to  category  3  =  227 
Number  of  documents  assigned  to  category  4  =  827 
Number  of  documents  assigned  to  category  6  =  54 
Number  of  documents  assigned  to  category  8  =  530 
Number  of  documents  assigned  to  category  9  =  169 
Number  of  documents  assigned  to  category  11  =  44 
Number  of  documents  assigned  to  category  12  =  3 
Number  of  documents  assigned  to  category  13  =  197 
Number  of  documents  assigned  to  category  14  =  140 
Number  of  documents  assigned  to  category  27  =  2506 
Number  of  documents  assigned  to  category  36  =  37 
Number  of  documents  assigned  to  category  48  =  78 
Number  of  documents  assigned  to  category  51  =  308 
Number  of  documents  assigned  to  category  52  =  70 
Number  of  documents  assigned  to  category  55  =  316 
Number  of  documents  assigned  to  category  56  =  646 
Number  of  documents  assigned  to  category  57  =  151 
Number  of  documents  assigned  to  category  60  =  191 
Number  of  documents  assigned  to  category  62  =  138 
Number  of  documents  assigned  to  category  65  =  6 
Number  of  documents  assigned  to  category  68  =  37 
Number  of  documents  assigned  to  category  69  =  52 
Number  of  documents  assigned  to  category  81  =  27 
Number  of  documents  assigned  to  category  82  =  67 
Number  of  documents  assigned  to  category  84  =  119 
Number  of  documents  assigned  to  category  86  =  55 
Number  of  documents  assigned  to  category  87  =  255 
Number  of  documents  assigned  to  category  88  =  34 
Number  of  documents  assigned  to  category  104  =  1 14 
Number  of  documents  assigned  to  category  107  =  28 
Number  of  documents  assigned  to  category  108  =  118 
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Number  of  documents  assigned  to  category  112  =  36 
Number  of  documents  assigned  to  category  1 13  =  124 
Number  of  documents  assigned  to  category  115  =  15 
Number  of  documents  assigned  to  category  1 16  =  80 
Number  of  documents  assigned  to  category  119=  18 
Number  of  documents  assigned  to  category  120  =  65 
Number  of  documents  assigned  to  category  122  =  22 
Number  of  documents  assigned  to  category  123  =  1 
Number  of  documents  assigned  to  category  124  =  3 
Number  of  documents  assigned  to  category  125  =  1 
Number  of  documents  assigned  to  category  130  =  33 
Number  of  documents  assigned  to  category  135  =  37 
Number  of  documents  assigned  to  category  137  =  79 
Number  of  documents  assigned  to  category  149  =  2 
Number  of  documents  assigned  to  category  150  =  10 
Number  of  documents  assigned  to  category  155  =  77 
Number  of  documents  assigned  to  category  157  =  639 
Number  of  documents  assigned  to  category  159  =  133 
Number  of  documents  assigned  to  category  160  =  29 
Number  of  documents  assigned  to  category  164  =  3 
Number  of  documents  assigned  to  category  165  =  1 
Number  of  documents  assigned  to  category  169  =  569 
Number  of  documents  assigned  to  category  172  =  71 
Number  of  documents  assigned  to  category  173  =  78 
Number  of  documents  assigned  to  category  177  =  32 
Number  of  documents  assigned  to  category  178  =  19 
Number  of  documents  assigned  to  category  179  =  27 
Number  of  documents  assigned  to  category  186  =  66 
Number  of  documents  assigned  to  category  187  =  1 
Number  of  documents  assigned  to  category  190  =  13 
Number  of  documents  assigned  to  category  192  =  63 
Number  of  documents  assigned  to  category  197  =  44 
Number  of  documents  assigned  to  category  202  =  15 
Number  of  documents  assigned  to  category  203  =  3 
Number  of  documents  assigned  to  category  205  =  84 
Number  of  documents  assigned  to  category  206  =  192 
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Number  of  documents  assigned  to  category  207  =  17 
Number  of  documents  assigned  to  category  22 1  =  29 
Number  of  documents  assigned  to  category  239  =  42 
Number  of  documents  assigned  to  category  242  =  25 
Number  of  documents  assigned  to  category  243  =  27 
Number  of  documents  assigned  to  category  247  =  12 
Number  of  documents  assigned  to  category  254  =  2 
Number  of  documents  assigned  to  category  255  =  4 
Number  of  documents  assigned  to  category  267  =  4 
Number  of  documents  assigned  to  category  280  =  6 
Number  of  documents  assigned  to  category  282  =  6 
Number  of  documents  assigned  to  category  312  =  7 
Number  of  documents  assigned  to  category  314  =  1 
Number  of  documents  assigned  to  category'  316  =  3 
Number  of  documents  assigned  to  category  318  =  16 
Number  of  documents  assigned  to  category  335  =  9 
Number  of  documents  assigned  to  category  339  =  3 
Number  of  documents  assigned  to  category  341  =  3 
Number  of  documents  assigned  to  category  342  =  3 
Number  of  documents  assigned  to  category  343  =  8 
Number  of  documents  assigned  to  category  344  =  9 
Number  of  documents  assigned  to  category  345  =  7 
Number  of  documents  assigned  to  category  347  =  12 
Number  of  documents  assigned  to  category  353  =  3 
Number  of  documents  assigned  to  category  354  =  1 
Number  of  documents  assigned  to  category  355  =  4 
Number  of  documents  assigned  to  category  358  =  21 
Number  of  documents  assigned  to  category  360  =  2 
Number  of  documents  assigned  to  category  367  =  1 
Number  of  documents  assigned  to  category  370  =  2 
Number  of  documents  assigned  to  category  376  =  3 
Number  of  documents  assigned  to  category  386  =  7 
Number  of  documents  assigned  to  category  432  =  2 
Number  of  documents  assigned  to  category'  455  =  1 
Number  of  documents  assigned  to  category  461  =  7 
Number  of  documents  assigned  to  category'  477  =  4 
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Number  of  documents  assigned  to  category  478  =  1 
Number  of  documents  assigned  to  category  503  =  8 
Number  of  documents  assigned  to  category  505  =  3 
Number  of  documents  assigned  to  category  507  =  3 
Number  of  documents  assigned  to  category  515  =  4 
Number  of  documents  assigned  to  category  516-2 
Number  of  documents  assigned  to  category  536  =  4 
Number  of  documents  assigned  to  category  539  =  1 
Number  of  documents  assigned  to  category  586  =  1 
Number  of  documents  assigned  to  category  592  -  3 
Number  of  documents  assigned  to  category  598  =  1 
Number  of  documents  assigned  to  category  619  -  2 
Number  of  documents  assigned  to  category  661  =  1 
Number  of  documents  assigned  to  category  668  =  1 
Number  of  documents  assigned  to  category  673  =  0 
Number  of  documents  assigned  to  category  674  =  0 
Number  of  documents  assigned  to  category  675  =  0 
Number  of  documents  assigned  to  category  676  =  0 
Number  of  documents  assigned  to  category  677  =  0 
Number  of  documents  assigned  to  category  678  =  0 
Number  of  documents  assigned  to  category  679  =  0 
Number  of  documents  assigned  to  category  680  =  0 
Number  of  documents  assigned  to  category  681  =  0 
Number  of  documents  assigned  to  category  682  =  0 
Number  of  documents  assigned  to  category  683  =  0 
Number  of  documents  assigned  to  category  684  =  0 
Number  of  documents  assigned  to  category  685  -  0 
Number  of  documents  assigned  to  category  686  =  0 
Number  of  documents  assigned  to  category  687  =  0 
Number  of  documents  assigned  to  category  688  =  0 
Number  of  documents  assigned  to  category  689  =  0 

Total  number  of  documents  with  category  assignments  =  14652 
Number  of  documents  with  topic(s)  assigned  =  1 1664 
Number  of  documents  with  no  assignments  at  all  =  10509 
Percentage  of  documents  categorized  =  66.08 
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Appendix  P 


List  of  categories 

Money/Foreign  Exchange  (MONEY-FX) 

Shipping  (SHIP) 

Interest  Rates  (INTEREST) 

**Economic  Indicator  Codes  (16) 

Balance  of  Payments  (BOP) 

Trade  (TRADE) 

Consumer  Price  Index  (CPI) 

Wholesale  Price  Index  (WPI) 

Unemployment  (JOBS) 

Industrial  Production  Index  (IPI) 

Capacity  Utilization  (CPU) 

Gross  National/Domestic  Product  ( GNP) 

Money  Supply  (MONEY-SUPPLY) 

Reserves  (RESERVES) 

Leading  Economic  Indicators  (LEI) 

Housing  Starts  (HOUSING) 

Personal  Income  (INCOME) 

Inventories  ( INVENTORIES ) 

Installment  Debt/Consumer  Credit  ( INSTAL-DEBT) 
Retail  Sales  (RETAIL) 

^Currency  Codes  (27) 

U.S.  Dollar  (DLR) 

Australian  Dollar  (AUSTDLR) 

Hong  Kong  Dollar  (HK) 

Singapore  Dollar  ( SINGDLR) 

New  Zealand  Dollar  (NZDLR) 

Canadian  Dollar  (CAN) 

Sterling  ( STG) 

D-Mark  ( DMK) 

Japanese  Yen  (YEN) 

Swiss  Franc  ( SFR) 

French  Franc  ( FFR) 

Belgian  Franc  ( BFR) 
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Netherlands  Guilder/Florin  (DFL) 

Italian  Lira  (LIT) 

Danish  Krone/Crown  (DKR) 

Norwegian  Krone/Crown  (NKR) 

Swedish  Krona/Crown  (SKR) 

Mexican  Peso  (MEXPESO) 

Brazilian  Cruzado  (CRUZADO) 

Argentine  Austral  (AUSTRAL) 

Saudi  Arabian  Riyal  (SAUDRIYAL) 

South  African  Rand  (RAND) 

Indonesian  Rupiah  (RUPIAH) 

Malaysian  Ringitt  (RINGGIT) 

Portuguese  Escudo  (ESCUDO) 

Spanish  Peseta  (PESETA) 

Greek  Drachma  (DRACHMA) 

**Corporate  Codes  (2) 
Mergers/Acquisitions  (ACQ) 

Earnings  and  Earnings  Forecasts  (EARN) 

**Commodity  Codes  (78) 

ALUM 

BARLEY 

CARCASS 

CASTOR-MEAL 

CASTOR-OIL 

CASTORSEED 

CITRUSPULP 

COCOA 

COCONUT-OIL 

COCONUT 

COFFEE 

COPPER 

COPRA-CAKE 

CORN-OIL 

CORN 

CORNGLUTENFEED 

COTTON 

COTTON-MEAL 

COTTON-OIL 
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COTTONSEED 

F-CATTLE 

FISHMEAL 

FLAXSEED 

GOLD 

GRAIN 

GROUNDNUT 

GROUNDNUT -MEAL 

GROUNDNUT-OIL 

IRON-STEEL 

LEAD 

LIN-MEAL 

LIN-OIL 

LINSEED 

LIVESTOCK 

L- CATTLE 

HOG 

LUMBER 

LUPIN 

MEAL-FEED 

NICKEL 

OAT 

OILSEED 

ORANGE 

PALLADIUM 

PALM-MEAL 

PALM-OIL 

PALMKERNEL 

PLATINUM 

PLYWOOD 

PORK-BELLY 

POTATO 

RAPE-MEAL 

RAPE-OIL 

RAPESEED 

RED-BEAN 

RICE 

RUBBER 

RYE 


SILK 
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SILVER 


SORGHUM 

SOY-MEAL 

SOY-OIL 

SOYBEAN 

STRATEGIC-METAL 

SUGAR 

SUN-MEAL 

SUN-OIL 

SUNSEED 

TAPIOCA 

TEA 

TIN 

TUNG-OIL 

TUNG 

VEG-OIL 

WHEAT 

WOOL 

ZINC 


**Energy  Codes  (9) 

Crude  Oil  (CRUDE) 

Heating  Oil/Gas  Oil  (HEAT) 
Fuel  Oil  (FUEL) 

Gasoline  (GAS) 

Natural  Gas  (NAT-GAS) 
Petro- Chemicals  (PET-CHEM) 
Propane  (PROPANE) 

Jet  and  Kerosene  (JET) 
Naphtha  (NAPHTHA) 
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Abstract 

The  primary  focus  of  this  project  was  the  development  of  a  Wmdows/PC-based  Interactive  Learning  Wall 
which  creates  a  virtual  blackboard  that  can  be  controlled  at  a  distance  by  a  classroom  instructor.  The  most 
important  component  of  the  system  is  a  wireless  mouse  emulator  that  is  implemented  using  a  software- 
controlled  standard  red  laser  pointer.  With  this  system  the  instructor  is  no  longer  tethered  to  the  computer 
console.  A  video  camera  looks  at  the  projected  screen  and  feeds  its  output  to  the  computer.  Our  programs 
detect  and  interpret  the  location  of  the  laser  beam,  map  it  to  screen  coordinates,  and  move  the  system 
mouse  cursor  accordingly,  thus  simulating  system  mouse  actions.  Using  a  unique  algorithm  to  correct  for 
the  geometry  between  the  camera  and  the  screen,  we  can  accurately  point,  click,  drag,  and  perform  other 
mouse  manipulation  actions  with  the  laser  pointer.  A  secondary  component  of  the  Interactive  Learning 
Wall  is  a  wireless  LAN  system  that  permits  communication  between  student  laptop  computers  and  the 
instructor’s  computer.  Using  this  system  and  programs  we  developed,  students  can  make  requests  from  their 
laptops  to  take  control  of  the  main  computer’s  pointer  de\ice  and  keyboard.  This  allows  them  to  interact 
directly  with  the  instructor’s  computer  in  order  to  perform  tasks  like  pointing  to  specific  parts  of  the 
presentation  for  clarification  of  doubts  and  sharing  of  information.  A  third  aspect  of  the  project  was  an 
evaluation  of  off-the-shelf  voice  command  recognition  software  which  would  enable  the  instructor  to 
control  his  presentation  with  voice  commands.  A  final  aspect  of  this  project  was  to  provide  an  environment 
in  which  both  graduate  and  undergraduate  students  could  work  as  a  team  on  a  real-world  human-computer 
interface  problem. 


THE  INTERACTIVE  LEARNING  WALL:  A  PC-BASED,  DEPLOYABLE  DATA  WALL  FOR  USE  IN  A 

COLLEGE  CLASSROOM 

Richard  R.  Eckert 


Introduction 

Teaching  and  learning  involve  the  effective  exchange  of  information  between  an  expert  and  a  group  of  novices.  This  kind 
of  activity  permits  the  novice  to  understand  and  perform  new  tasks.  Although  traditionally  we  think  of  teaching  and 
learning  as  occurring  in  a  school  setting,  the  concepts  can  be  applied  to  any  realms  of  human  endeavor-including  the 
military.  In  that  realm,  the  effective  exchange  of  information  is  particularly  important. 

Some  of  the  most  effective  teaching  takes  place  in  one-on-one  or  small  group  settings.  This  kind  of  environment  provides 
for  individual  attention  and  an  easy,  non-intimidating  rapport  between  student  and  teacher  that  is  conducive  to  learning. 
As  the  material  to  be  learned  becomes  more  complex  and  voluminous,  the  number  of  learners  (novices)  becomes  much 
greater  than  the  number  of  teachers  (experts),  so  that  the  small-group  setting  becomes  impractical.  To  facilitate  the 
presentation  to  larger  student  groups,  the  teacher  must  have  access  to  some  sort  of  display  device  to  post  lessons  and/or 
notes.  Artifacts  such  as  blackboards,  whiteboards,  overhead  projection  systems,  and  computerized  projection  systems  have 
become  very  prevalent.  Unfortunately,  in  large-group  settings  that  use  this  kind  of  equipment,  interaction  between  the 
teacher  and  individual  students  is  kept  to  a  minimum  at  least  partly  because  of  the  fact  that  the  teacher  is  tethered  to  the 
display  system. 

Computers  have  added  another  dimension  to  today’s  classroom  by  greatly  extending  the  type  of  data  that  can  be 
experienced  by  students  and  teachers.  Multimedia  technology  has  provided  interactive  applications,  lessons,  and 
presentations  that  capture  the  audience.  Slide  show  software  such  as  Microsoft  PowerPoint™  has  been  used  in  professional 
and  academic  presentations  and  provides  stunningly  attractive  artistic  layouts.  Complex  three-dimensional  graphics  can 
easily  be  visualized  and  manipulated.  Teleconferencing  can  bring  experts  into  the  classroom  and  reduce  the  immense  cost 
normally  associated  with  guest  speakers.  The  World  Wide  Web  has  provided  a  window  into  the  universe-an  infinite  source 
of  learning  materials— that  is  now  becoming  accessible  to  classrooms.  Unfortunately  many  faculty  members  do  not  feel 
comfortable  with  the  vast  array  of  equipment  they  are  confronted  with,  and,  hence,  do  not  use  it  willingly .  Making  use  of 
these  devices  more  natural,  intuitive,  and  interactive  has  thus  become  very  important. 

The  Interactive  Data  Wall 

The  Advanced  Displays  and  Intelligent  Interfaces  group  at  the  Rome,  NY  Air  Force  Research  Laboratory  has  created  a 
sophisticated  and  expensive  presentation  system  that  involves  many  of  the  above-mentioned  components.  Its  Interactive 
DataWall"  consists  of  an  ultra-high-resolution  wall  display  driven  by  an  SGI  ONYX  computer  with  various  types  of 
wireless  input  devices  shared  among  multiple  users.  The  devices  include:  speech  interaction,  camera-tracked  laser  pointers, 
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and  flight  sticks.  The  purpose  of  the  DataWall  is  to  display  large  amounts  of  information  in  a  contiguous  area  and  to 
provide  multiple  users  with  an  intuitive  and  unencumbered  means  of  accessing  and  manipulating  the  data.  [1] 


In  this  project,  two  graduate  students,  six  undergraduates  and  the  project  director  were  able  to  develop  an  Interacts  e 
Learning  Wall.  This  is  a  scaled-down,  PCAVindows-based  implementation  of  the  Rome  Laboratory  facility  designed  for 
use  in  a  civilian  environment  More  precisely,  the  Learning  Wall  adapts  the  Air  Force  technology  to  benefit  a  modem 
classroom.  It  should  also  be  pointed  out  that  its  successful  implementation  can  be  of  benefit  to  the  Air  Force,  since  with  an 
inexpensive  PC,  wireless  networking  capability,  a  readily-available  RGB  projection  system,  an  inexpensive  video  camera, 
and  a  laser  pointer,  decision  makers  can  effectively  exchange  information  and  have  access  to  the  world. 

The  Interactive  Learning  Wall 

Figure  1  shows  the  interactive  learning  wall  setup. 


Figure  1.  Rear  Screen  Learning  Wall  Setup. 

Although  a  rear-screen  configuration  is  shown,  the  system  can  also  be  used  with  the  projector  and  camera  in  front  of  the 
projection  screen.  It  is  important,  however,  in  a  front-screen  setup  that  the  operator  not  shadow  the  image  being  display  ed 
by  the  projector  and  viewed  by  the  camera.  The  projector  projects  whatever  content  the  professor  is  displaying  on  the 
screen.  This  could  be  a  PowerPoint™  slide  presentation  or  any  other  visual  material  from  the  computer.  The  professor  uses 
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the  laser  pointer  to  point  at  and  interact  with  the  material  on  the  screen.  The  interaction  is  very  reminiscent  of  that  obtained 
with  a  standard  mouse.  The  camera  detects  the  bright  laser  spot  on  the  screen,  and  our  software  computes  the  corresponding 
location  on  the  screen  and  moves  a  "mouse  cursor"  icon  accordingly. 

In  our  implementation,  the  lower  right-hand  comer  of  the  screen  displays  a  white  box  that  shows  the  current  mode  of 
operation.  Upon  startup  it  says  LEFT.  This  means  that  if  the  user  “clicks”  with  the  laser  pointer,  it  generates  the  same 
effect  that  hitting  the  left  mouse  button  would  generate.  The  imerface  is  similar  to  that  used  with  touch  pads  on  some  laptop 
computers.  The  first  time  the  user  directs  the  laser  pointer  at  the  screen  it  is  treated  as  a  location  identifier.  In  other  words 
the  user  needs  to  accurately  point  at  the  location  where  he  or  she  would  like  to  push  the  button.  To  generate  a  single  click 
the  user  releases  the  pointer,  and  within  one  second  in  close  proximity  to  the  previous  location,  presses  the  pen  again.  To 
generate  a  double  click  this  process  is  repeated.  Although  this  may  seem  like  an  odd  implementation,  to  achie\  e  accuracy  at 
a  distance  it  is  essential  to  take  into  account  the  jitter  of  a  user's  hand.  In  our  implementation,  the  first  tune  the  laser  point  is 
seen  is  used  as  an  "aim"  and  the  second  is  a  "fire." 

The  other  modes  of  operation  are  RIGHT  and  DRAW,  and  they  are  selected  by  turning  off  the  laser  pointer  within  the 
white  box  in  the  lower  right-hand  comer  of  the  screen.  In  RIGHT  mode  the  program  generates  right  mouse  button  clicks. 
In  DRAW  mode  a  color  bar  appears  to  the  right  of  the  screen  and  displays  four  colors;  yellow,  red.  blue  and  green.  The 
active  color  is  highlighted  by  a  white  border.  To  draw  the  user  only  needs  to  click  once  and  the  program  will  continue  to 
draw  until  the  laser  pointer  is  released.  In  order  to  switch  colors  the  user  releases  the  laser  pointer  over  the  desired  color. 

The  computer 

Since  real-time  digital  video  processing  is  required,  a  fairly  well-endowed  computer  with  a  video  capture  card  is  necessary 
to  allow  implementation  of  the  Learning  Wall.  The  system  also  has  to  simultaneously  handle  user,  system,  and  application 
events.  We  used  a  computer  with  two  Intel  Pentium  II  300  MHz  processors,  128  megabytes  of  SDRAM,  an  IBM  8  gigabyte 
ultra  wide  SCSI  hard  drive,  an  8  megabyte  ATI  Expert@Work  2x  AGP  video  card,  and  a  Computer  Eyes/PCI  video  capture 
card.  This  was  much  more  than  adequate.  Tests  we  performed  determined  that  the  maximum  utilization  of  the  processor 
was  only  17  percent  while  running  the  Learning  Wall's  laser  pointer  software. 

The  Learning  Wall :  Laser  Pointer  Software 

The  software  to  run  the  system  is  broken  into  three  pieces.  The  first  and  most  hardware-related  software  component 
DETECT,  performs  low-level  detection  of  the  laser  pointer  in  the  camera's  viewable  area.  There  are  two  other  applications 
that  use  the  data  provided  by  this  program:  INIT  during  the  initialization  process  and  LASERPEN  during  normal  operation 
Both  of  these  helper  applications  run  with  a  window  title  of  LASERXY.  All  of  the  programs  ask  Windows  to  register  a 
WM  LASER  message  through  which  communication  takes  place.  All  applications  that  request  a  message  with  that  title 
will  be  returned  the  same  unique  identifier.  Figures  2  and  3  illustrate  the  relationships  between  these  programs. 


r 

]  FILES: 

j  nrrrx.Dvr 
;  DUTY.  DM 

t 

4 - 

ntrr.EXE 

(VlBdmr 

titled 

ULSBOOf) 

4 - 

DETECT. EMC 

Translated  coordinates  »<_LXSER 


Figure  2.  The  Interaction  between  DETECT  and  INIT. 
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Figure  3.  The  Interaction  between  DETECT  and  LASERPEN. 


The  camera  captures  individual  frames  at  320  pixels  in  the  x  dimension  and  240  pixels  in  the  y  dimension  at  as  close  to  30 
frames  per  second  as  possible.  Depending  on  how  taxed  the  system  is  from  user  commands  the  frame  rate  can  change. 
Each  individual  frame  is  processed  to  see  if  there  was  an  instance  of  a  pixel  being  brighter  than  a  certain  threshold.  The 
environment  that  is  present  determines  this  threshold.  DETECT  can  perform  some  functions  as  a  stand-alone  program,  but 
it  also  cooperates  with  other  applications  through  Windows  messages.  As  a  stand-alone  and  cooperative  application  it  can 
display  the  output  of  the  camera  and  also  provide  the  tracking  information  of  the  laser  pointer  in  camera  coordinates.  To 
improve  speed,  the  tracking  is  performed  by  discarding  half  of  the  actual  pixels  that  are  visible  to  the  camera.  After  an 
instance  of  a  pixel  over  the  threshold  is  found  DETECT  discerns  if  the  pixel  that  was  most  recendy  skipped  had  levels  of 
red,  green  and  blue  greater  than  the  threshold  If  so,  the  coordinates  of  the  skipped  pixel  are  used.  This  provides  the 
accuracy  of  the  entire  resolution  of  the  camera  along  with  the  speed  of  checking  only  half  the  pixels  initially.  Also,  to 
improve  efficiency,  the  program  stops  looking  for  an  instance  of  the  laser  in  a  particular  frame  once  it  is  seen  In  order  to 
run  as  a  cooperative  application,  DETECT  asks  Windows  to  return  the  handle  to  LASERXY.  the  program  that  spawned  it. 
This  allows  DETECT  to  send  Window’s  messages  to  that  specific  application  instead  of  a  WM_BROADCAST  message  that 
could  severely  slow  down  the  system.  Once  the  camera  distinguishes  an  instance  of  the  laser  pointer,  it  then  sends  messages 
to  either  LASERPEN  or  INIT  with  the  laser  pointer's  position  in  the  frame. 

Sending  the  appropriate  message  involves  knowing  information  about  the  previous  frame  as  well.  If  in  the  previous  frame 
an  instance  of  the  laser  pointer  wasn’t  seen,  the  program  sends  the  coordinates  along  with  a  first  seen  flag.  If  it  was  seen  in 
the  last  frame,  then  it  sends  a  continue_seen  flag.  When  the  laser  pointer  is  not  seen,  but  previously  it  was  seen,  the 
program  sends  (0,0)  for  the  camera  coordinates  and  a  flag  of  not_seen.  To  minimize  the  number  of  Windows  messages,  the 
application  doesn't  continually  send  not_seea  messages.  Figure  4  is  a  state  diagram  showing  the  message  states  of 
DETECT.  Figure  5  is  a  flow  chart  showing  the  logic  involved  in  DETECT 
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seen  /  no  message 


$*en  /  first_seen 
message  and  (x,y) 


i‘ot  seen  /  not_seen 
sassage  and  (0,0) 


Seen  /  contime_seen 
message  and  (x,y) 


Figure  4.  DETECT  Message  States. 


Figure  5.  A  High-Level  Flow  Chart  of  the  DETECT  Program. 


Menu  Choices 


If  DETECT,  miming  as  a  stand-alone  program,  wasn’t  spawned  by  an  application  with  the  title  of  LASERXY ,  then  it  starts 
up  in  a  normal  window,  displaying  the  captured  frames  along  with  the  coordinates  of  the  point  where  the  laser  pointer  was 
seen.  This  can  be  toggled  on  and  off  by  going  to  the  Image  menu  and  selecting  Toggle  Output.  If  an  instance  of  the  either 
INIT  or  LASERPEN  is  running,  then  the  program  toggles  the  output  off  and  minimizes  itself.  The  default  camera  position 
setting  is  front  screen  and  can  be  toggled  under  the  Image  menu  as  well.  While  using  the  program  for  detection  of  the  laser 
pointer,  it  must  capture  at  320  pixels  by  240  pixels  but  for  use  as  a  stand-alone  application,  under  Image/Preview  Size,  the 
user  has  many  resolution  options:  80x60,  160x120,  320x240,  480x360,  560x420  and  640x480. 
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Camera  to  Screen  Transformation  (1NIT) 


In  a  perfect  environment  the  camera  would  be  inside  the  projector  and  see  exactly  as  much  as  the  projector.  In  addition,  the 
lens  would  be  perfectly  flat  so  that  every  pixel  is  the  same  size  as  every  other  pixel.  Unfortunately  this  is  not  the  case,  so 
there  is  a  need  to  tell  LASERXY  some  specific  information  about  the  geometry  of  the  camera  and  the  image  being 
projected.  This  is  done  through  an  initialization  procedure  in  which  two  functions  that  produce  a  discrete  display  coordinate 
for  every  \isible  camera  coordinate  are  generated: 

x  display  coordinate  =  Fx  (x,y  camera  coordinate) 
y  display  coordinate  =  Fy  (x,y  camera  coordinate) 

The  resulting  transformation  data  is  then  saved  onto  disk  for  LASERPEN  to  read. 

Upon  running  INIT  the  arrow  icon  is  replaced  with  a  series  of  four  targets  toward  which  the  user  should  point  the  laser. 
These  target  points  are  in  the  four  comers  of  the  screen.  Since  the  user’s  hand  is  not  perfectly  steady  the  software  expects 
the  user  to  point  at  each  target  and  release  the  pointer  when  on  target.  The  point  w  here  the  laser  was  last  seen  is  taken  as  the 
initialization  point.  The  resulting  four  points  are  stored  and  used  in  the  camera-to-screen  transformation.  What  the  camera 
sees  of  the  projected  image  and  the  four  target  points  (PI,  P2,  P3,  P4)  could  be  similar  to  what  is  shown  in  Figure  6.  The 
projected  image  is  640  by  480  pixels,  but  in  terms  of  camera  viewable  coordinates  it  is  only  320  by  240  pixels. 


240 

480 


| - 320  - 1 

- 648 - 

Figure  6.  Simulated  Camera  Viewable  Area. 

First  the  intersection  points  of  the  "vertical"  and  "horizontal"  lines  bounding  the  area  enclosed  by  the  four  target  points  are 
computed  using  the  parametric  equations  for  straight  lines.  Any  point  within  the  camera’s  viewable  area  is  converted  to  its 
scaled  screen  coordinates  by  projecting  a  line  from  the  point  to  each  of  the  two  pre\iously  calculated  intersection  points.  As 
shown  in  Figure  7,  those  lines  will  each  intersect  with  an  opposing  line.  The  intersection  points  are  then  stored  and  the 
result  is  evaluated  as  a  percentage  of  the  overall  line  and  linearly  scaled.  In  this  example,  the  x  values  are  scaled  as 
a  percentage  of  640  pixels  and  the  y  values  are  scaled  as  a  percentage  of  480  pixels  This  information  is  calculated  for  every' 
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point  in  the  camera’s  viewable  area  and  is  stored  in  two  two-dnnensional  arrays,  one  for  x  coordinate  values  Hid  one  for  y 
coordinate  values.  At  the  end  of  the  initialization  program,  the  wo  arrays  are  stored  onto  the  disk.  This  ensures  that  once  a 
good  initialization  is  calculated,  it  will  never  have  to  be  repeated  until  there  is  a  physical  change  in  the  setup. 


Figure  7.  Camera  to  Screen  Transformation. 


The  previous  example  dealt  with  displayed  image  resolutions  of  only  640  by  480  pixels.  In  practice  the  software  can  handle 
dynamic  changes  in  resolutioa  Instead  of  calculating  the  poims  in  reference  to  a  display  of  640  by  480  pixels,  it  calculates 
the  points  in  reference  to  a  display  of  512,000  by  288,000  pixels.  This  is  a  common  factor  that  can  be  evenly  divided  by 
640  by  480  pixels,  800  by  600  pixels  and  1024  by  768  pixels.  By  computing  the  points  in  this  fashion  the  program  doesn  t 
have  to  know  the  resolution  of  the  actual  display,  making  it  much  more  general  and  useful.  This  also  allows  calibration  to 
occur  regardless  of  whether  the  resolution  of  the  screen  is  the  same  during  calibration  and  normal  operations.  Figure  8  on 
the  next  page  is  a  flow  chart  showing  how  y  screen  coordinates  are  computed  from  camera  coordinates  in  640  X  480 

display  mode. 


LASERPEN 

The  final  application  that  works  directly  with  the  laser  pointer  is  called  LASERPEN.  This  program  is  the  meat  and  bones  of 
the  user  interface  since  it  listens  to  Windows  messages  and  moves  the  pointer  accordingly.  The  program  begins  by  opening 
the  files  that  were  created  by  INIT  and  stores  them  into  two  arrays,  one  for  the  x-coordinates  and  one  for  the  v-coordinates. 
After  loading  the  files,  a  correction  must  be  performed  to  change  from  the  general  x  and  y  values  to  the  x  and  y  values  for 
the  current  resolution.  For  resolutions  of  640  by  480  pixels,  the  x  and  y  values  are  divided  by  800  and  610.  respectively; 
for  resolutions  of  800  by  600,  they  are  divided  by  640  and  4S0;  for  resolutions  of  1024  by  768,  they  are  div.ded  by  oOO  and 
375.  When  a  resolution  change  occurs,  the  program  reloads  the  files  from  the  disk  and  re-performs  these  calculations.  By 
doing  the  calculations  only  once  for  each  resolution  change,  the  program  removes  two  divisions  that  would  otherwise  need 
to  be  performed  at  every  instance  of  the  laser  pointer  inside  the  body  of  the  loop. 


Figure  8.  A  High-Level  Flow  Chart  Showing  how  the  y  Screen  Coordinates  are  Computed 
For  Each  point  in  the  Camera's  Viewing  Area. 
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LASERPEN  doesn’t  open  a  standard  window  like  most  applications.  Instead  it  creates  a  visible  popup  window  in  the  lower 
right  hand  comer  of  the  screen  that  is  25  pixels  high  and  50  pixels  wide.  This  window  has  no  borders,  is  solid  white,  and 
cannot  be  overwritten  by  other  apphcatron  windows.  It  is  used  to  display  rile  cmienl  emulation  mode  of  ihe  laser  poinler 

(LEFT,  RIGHT,  or  DRAW). 

Windows  and  Visual  C++  5.0  provide  functions  that  make  generating  button  clicks  fairly  simple.  The  function 
mouse_event()  takes  as  parameters  the  desired  button  to  press  and  the  location.  In  order  to  be  able  to  generate  dounle  clicks, 
the  mouse  sensitivity  under  Windows  needs  to  be  lowered  using  the  Windows  Control  Panel  Mouse  settings. 


To  alleviate  the  problem  of  user  unsteadiness  the  following  approach  is  taken:  When  the  laser  is  turned  on,  an  imaginary 
area  equivalent  to  1/16  of  the  overall  horizontal  resolution  is  drawn  around  the  point.  If  the  laser  is  detected  again  within 
one  second  of  the  last  time  it  was  detected  and  falls  within  this  imaginary  area,  then  LASERPEN  generates  the  proper 
action.  In  LEFT  mode  a  left  mouse  button  mouse  button  down  message  is  generated  and,  only  when  the  pointer  is  not 
detected,  is  a  left  mouse  button  up  message  generated.  In  the  RIGHT  mode  the  same  actions  are  performed  to  with  the 
generation  of  right  mouse  button  messages.  If  the  program  is  in  DRAW  mode,  instead  of  generating  a  button  click,  laser 
detection  is  used  to  generate  the  endpoint  of  a  line  segment.  While  the  program  still  sees  the  laser  pointer,  the  Ice  is  drawn 
to  that  point  from  its  previous  location.  In  all  modes  of  operation  the  mouse  pointer  is  instantly  moved  to  its  nr*  location. 
This  is  much  different  from  the  case  of  a  standard  mouse  in  which  the  system  has  to  take  into  account  its  current  location 


before  it  can  begin  moving  the  mouse  cursor.  Figure  9  on  the  next  page  shows  a  high-level  flow  chart  of  the  logic  of  the 


LASERPEN  application. 


All  the  required  communication  between  these  programs  is  done  with  WM_LASER  messages.  As  seen  m  Figures  3  and  4, 
LASERPEN  receives  the  WM_LASER  message  from  DETECT.  This  message  includes  the  relative  location  and  the 
first_seen,  continue_seen,  and  not.seen  flags.  When  LASERPEN  receives  this  message,  a  lookup  is  performed  into  the  x 
and  y  coordinate  arrays  and  the  mouse  is  moved  to  that  new  location  using  the  Windows  function  SetCursorPosn 


Thp  Wireless  LAN  Component 


One  way  of  enabling  student  laptop  computers  to  communicate  with  the  instructor's  computer  is  to  set  up  a  wireless  local 
area  network  (WLAN).  Our  WLAN  consists  of  a  Lucent  Technologies  WavePOINT  II  base  station  connected  %  a  a  10base2 
cable  to  the  server  machine.  The  laptops  then  use  type  2  slot  PCMCIA  WaveLAN  cards  with  attached  antennas  tot  stick  to 
the  back  side  of  the  laptop.  The  base  stations  and  the  PCMCIA  cards  auto-connect  when  they  are  in  proximity  ci  each  other 
and  provide  the  laptops  with  access  to  the  Intranet  that  the  server  provides.  The  LAN  can  provide  up  to  4  Mbps  bandwidth 
broadcast  at  2.4  GHz  split  among  all  users  in  the  range  of  the  base  station.  In  an  open  area  such  as  Binghamton  University’s 
lecture  hall  12,  the  connectivity  is  limited  to  200m,  which  is  more  than  enough  for  even  our  largest  lecture  bill.  The  only 
requirement  that  the  base  station  has  is  that  it  be  mounted  in  plain  view  on  the  wall  and  that  its  antenna  face  tie  classroom. 
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FALSE 


If  VVM  LASER  message 


G  et  the  corr  ent  tun  e. 
d  time  *  the  last  time  -  the  current  time. 

Move  the  window  to  the  top. 

Set  the  imaginary  area  =  1/16  horizontal 
resolution. 


TRUE 


FALSE 


If  the  m  es  sage  c  ontaine  i  a 
Continue  Seenflag 


Set  pressed  *  2. 

Move  the  mouse  cursor  to  the 
new  location. 

Store  this  location 


If  the  message  contained  a 
Fir  stJSeen  flag 


Ifdjtime  <  1  second  and  the 
new  location  is  in  the  imaginary 


f  D  RArV  m  ode  and  pre 


There  are  no  software  modifications  that  have  to  be  made  to  exists  TCP/IP  based  applications  as  the  Wireless  LAN  is 
merely  an  implementation  that  is  transparent  to  the  Operating  System 

In  order  to  allow  students  to  control  the  instructor’s  mouse  and  keybcard  over  the  WLAN,  we  created  two  custom  pieces  of 
Windows  software:  a  server  application  (CLICKSERVE)  and  a  cliem  application  (CLICKCLIENT)  for  each  student  laptop. 
These  are  Microsoft  Visual  C++,  MFC,  dialog-based  applications.  When  a  student  starts  his  laptop  computer,  the  wireless 
LAN  will  automatically  connect  the  user  to  the  intranet  and  provide  him  with  an  IP  address  for  the  TCP/EP  protocol.  This 
gives  the  student  all  of  the  facilities  required  to  run  nearly  every  web  and  Internet  based  applicatioa  If  a  student,  connected 
to  the  network,  runs  our  CLICKCLIENT  application  from  his  laptro,  he  can  control  the  instructor  computer's  cursor  and 
keyboard,  thereby  gaining  control  of  the  system  console  just  as  if  t  were  his  laptop.  This  type  of  interaction  allows  easy 
synchronization  of  students’  questions  to  presentation  material  by  allowing  the  students  to  easily  point  to  the  area  in 
question.  This  also  allows  the  students  to  change  between  slides,  internet  with  software  that  is  typically  single  user,  and  thus 

learn  more  efficiently. 

Starting  CLICKSERVE  presents  the  instructor  with  a  minimi zaKe  window  that  gr.es  him  the  option  of  accepting  or 
denying  connections  and  allowing  or  denying  student  control  over  the  system  mouse  and  keyboard.  This  allows  the 
instructor  the  flexibility  of  being  able  to  decide  w  hen  student  intervention  is  most  convenient  or  warranted. 


When  the  student  runs  CLICKCLIENT,  he  is  required  to  login  to  tie  server  using  his  own  unique  username  and  password. 
This  allows  the  server  application  to  have  more  knowledge  of  the  incoming  requests  and  could  be  made  customizable  for 
different  people.  After  pressing  the  “Start  Capture’*  button,  the  student  computer's  screen  is  blanked  white,  and  any  mouse 
or  keyboard  actions  he  takes  will  be  reflected  on  the  instructor's  computer  screen.  (The  instructor  also  has  the  capability  of 
controlling  his  own  mouse  cursor;  mouse/key  board  messages  from  the  instructor's  system  take  precedence  over  messages 
coming  over  the  local  network  from  a  student  computer.)  Pressmg  the  <ESC>  key  on  the  student's  laptop  disables  the 
student  from  controlling  the  system  mouse  and  returns  his  screen  to  normal.  CLICKCLIENT  also  allows  the  student,  to 
“electronically”  raise  his  or  her  hand  Pressing  the  <F1>  "page"  key  on  the  laptop  causes  an  audio  file  to  be  play  ed  on  the 
server  to  notify  the  instructor  that  a  student  has  a  question.  CLICKCLIENT  and  CLICKSER\*E  communicate  through 
standard  TCP/IP  sockets.  Whenever  the  user  of  the  student  computer  performs  any  mouse  or  keyboard  actions, 
CLICKCLIENT  sends  the  appropriate  Windows  message  aid  data  parameters  over  the  socket  connection  to 
CLICKSERVE. 


Some  of  the  mouse  messages  of  interest  are: 


WM_*BUTTONDOWN 
WM_*BUTTONUP 
WM_*BUTTONDBLCLK 
WM_MOU  SEMO  VE 


//  A  mouse  button  was  pressed  (*  =  L,  R,  or  M) 

//  A  mouse  button  was  released  (*  =  L,  R,  or  M) 

//  A  mouse  button  was  double  clicked  (*  =  L,  R  or  M) 
//  User  mov  ed  the  mouse 
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The  data  that  is  sent  over  the  network  connection  is  identical  to  the  information  that  is  packaged  in  a  normal  Windows 
mouse  event  message-namely  the  x,  y  coordinates  of  the  mouse  cursor  and  codes  indicating  what  action  occurred  and  the 
status  of  the  mouse  buttons  and  certain  keyboard  keys  (Shift,  Ctrl,  Alt)  when  the  event  occurred 


The  keyboard  events  of  interest  are: 

WMKEYDOWN  //  User  pressed  a  kevboard  key 

WM_KEYUP  //  User  released  a  keyboard  key 

The  data  that  is  sent  over  the  network  for  these  kinds  of  events  consists  of  the  virtual  key  code  that  identifies  the  key 
pressed  or  released  and  other  information  related  to  the  keyboard  event  that  triggered  the  message. 


Finally  if  the  student  presses  his  <F1>  key  (thereby  attempting  to  "page"  the  instructor),  the  message  is  received  as: 
WM_USER+100  //  User  pressed  his  "page"  key 

The  response  of  CLICKSERVE  is  to  play  a  wave  file  on  the  instructor's  computer. 

The  connection  and  communication  between  the  student  and  instructor  computers  as  well  as  the  salient  features  of  the 
CLICKCLIENT  and  CLICKSERVE  programs  are  shown  in  Figure  10  on  the  next  page. 

By  having  the  client  send  the  information  blindly  and  pass  it  on  to  Windows  functions  in  the  server  that  accept  these 
parameters,  the  CLICKCLIENT  and  CLICKSERVE  programs  are  general  enough  to  sustain  significant  changes  in  the  way 
Windows  processes  and  packages  mouse  and  keyboard  events. 

The  Voice  Communications  Component 


In  order  to  add  a  further  degree  of  freedom  to  the  instructor,  we  attempted  to  add  a  voice  command  recognition  component 
to  the  learning  wall.  The  idea  would  be  for  the  instructor  to  be  able  to  control  his  application  by  voice  commands  over  a 
wireless  microphone  as  well  as  by  using  the  laser  pointer.  We  had  hoped  to  be  able  to  use  an  off-the-shelf  speech 
recognition  software  package  to  control  the  kinds  of  applications  an  instructor  would  be  using  in  the  classroom  (e  g.,  a 
Powerpoint  presentation).  We  performed  experiments  with  Dragon  Systems  Dragon  Dictate  for  Windows  and  IBM  s  Via 
Voice,  but,  in  both  cases,  ran  into  reliability  problems.  It  seems  that  the  kinds  of  voice  recognition  packages  that  are 
currently  available  are  not  capable  of  giving  a  sufficiently  high  percentage  of  successful  commands  recognized  to  be  really 
useful  in  the  kind  of  "noisy"  classroom  environment  where  we  envision  the  learning  wall  would  be  used.  A  great  deal  of 
further  research  needs  to  be  done  in  this  area. 
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Figure  10.  CLICKCLIEHT  and  CLICKSERVE  Controlling  Windows  Messages  Over  the  BLAH  Connection. 
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The  Students 


Because  of  the  AFOSR  grant,  we  were  able  to  provide  a  real  opportunity  to  do  research  in  human-computer  interfaces  to 
two  graduate  students  (Jason  Moore  and  Veersen  Ghatge)  and  six  undergraduate  students  (Jaron  Rubenstein,  Richard  Lao, 
Steaphan  Greene,  Brian  Lalor,  Thomas  Happ,  and  Justin  Terpstra).  Tnis  group  of  students  worked  very  well  as  a  team  and 
were  able  to  successfully  install,  debug,  and  maintain  all  of  the  hardware  we  used  in  the  project.  They  also  designed,  wrote, 
and  debugged  all  of  the  related  software.  The  graduate  student  who  did  the  most  of  the  work  on  the  design  and 
implementation  of  the  laser  pointer  system,  (Jason  Moore)  received  his  M.S.  in  Computer  Science  from  Binghamton 
University  in  May,  1993,  is  now  working  for  the  Air  Force  Laboratory  at  Rome,  New  York  on  a  project  that  is  very  closely 
related  to  our  learning  wall  project.  Eventually  he  is  planning  to  continue  studies  toward  the  Ph  D  degree  at  Binghamton 
University  in  the  area  of  human-computer  interaction.  One  of  the  undergraduate  students  involved  in  the  project  (Steaphan 
Greene)  worked  last  summer  at  Rome  Lab.  He  and  one  of  the  undergraduates  who  joined  our  group  this  semester  Justin 
Terpstra)  are  both  considering  working  at  Rome  Lab  this  coming  summer. 

Possible  Publications 

We  are  in  the  process  of  WTiting  papers  describing  our  results.  Our  plan  is  to  try  to  submit  them  to  one  of  the  ACM’s 
SIGCHI  (Special  Interest  Group  on  Computer  Human  Interaction)  publications.  One  of  the  papers  will  be  on  the  laser 
pointer  system  and  the  other  on  the  CLICKCLIENT/CLICKSERVE  wireless  LAN  system. 

Conclusions 


At  Binghamton  University  several  people  have  tried  using  our  Intemctive  Learning  Wall  system.  In  all  cases  the  learning 
curve  required  to  become  productive  was  very  small.  The  time  needed  to  become  proficient  was  spent  mostly  in  becoming 
familiar  with  how  the  laser  pointers  "click"  sequence  wrorks.  After  becoming  comfortable  with  that,  the  interface  preved  to 
be  quite  intuitive.  On  the  basis  of  our  initial  tests,  we  feel  that  mis  type  of  device  could  be  used  effectively  a:  other 
institutions. 
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Abstract 

The  multi-user  virtual  environment  is  a  computer-generated  synthetic  world  on 
the  network  where  users  can  navigate  and  interact  with  each  other  in  the  same  virtual 
world.  The  World  Wide  Web  (WWW)  environment  is  a  client/server  architecture 
composed  ol  various  server  sites  to  form  a  global  database  web  over  the  Internet. 
Since  the  existing  WWW  browsers  have  difficulty  to  create  an  interactive  multi-user 
virtual  world,  the  Shared  Web  system,  a  three-dimensional  web  browsing  system,  is 
designed  to  support  the  interactions  among  the  clients  over  the  WWW  environment. 
This  report  describes  the  design  and  implementation  of  SharedWeb. 

The  SharedWeb  system  has  the  following  features  that  are  necessary  for  the 
seamless  integration  between  a  multi-user  interactive  system  and  the  WWW 
environment.  1 .  The  user  can  download  a  scene  file  from  any  WWW  server  with  the 
HyperText  Transfer  Protocol  (HTTP).  2.  The  SharedWeb  system  provides  the 
hyperlink  feature  to  retrieve  various  media  resources  that  are  supported  by  the  WWW 
environment.  3.  I  he  SharedWeb  system  can  handle  information  exchange  between 
HI  ML  documents  and  the  objects  inside  a  virtual  world.  4.  The  SharedWeb  system 
server  itsell  is  an  add-on  of  an  existing  WWW  server.  5.  The  user  can  directly  enter  a 
virtual  world  through  the  WWW  server  from  which  the  scene  file  is  downloaded. 

The  SharedWeb  system  has  been  successfully  tested  in  both  the  Local  Area 
Network  (LAN)  and  Wide  Area  Network  (WAN)  environment.  The  SharedWeb 
system  uses  the  Dead  Reckoning  technique  to  reduce  the  network  traffic  created  by 
the  interactions  among  the  users,  and  to  maintain  the  interactivity  even  when  the 
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congested  Internet  delays  the  PDUs.  After  receiving  a  delayed  PDU,  proper  delay 
compensation  and  smoothing  are  needed  to  make  the  interaction  realistic. 

1.  Introduction 

Virtual  environment  is  a  technique  to  immerse  the  user  into  a  synthetic  world 
built  by  the  computer  or  a  remote  environment  electronically  transmitted  to  the 
computer  for  the  purpose  of  training,  simulation,  prototyping,  design,  or  data 
acquisition.  Due  to  the  rapid  development  of  the  virtual  environment  technique,  the 
multiple-user  virtual  environment  has  become  a  popular  topic  over  the  years.  The 
multi-user  virtual  environment  is  a  computer-generated  synthetic  world  on  the 
network  where  users  can  navigate  and  interact  with  each  other  in  the  same  virtual 
world.  The  gaming  industry  has  developed  many  multi-user  systems  over  the 
Internet.  An  extensive  survey  of  these  existing  systems  can  be  found  in  Section  2. 

The  World  Wide  Web  (WWW)  is  a  project  that  brings  a  global  information 
universe  to  the  existence.  [1]  The  Virtual  Reality  Modeling  Language  (VRML) 
project  is  an  attempt  to  deliver  a  virtual  world  into  the  WWW  environment.  [2]  The 
original  goal  of  designing  VRML  language  is  to  allow  multiple  participants  to  interact 
over  the  Internet.  After  the  introduction  of  the  VRML  language,  many  VRML- 
supported  browsers  were  developed,  which  include  the  WebSpace  from  SGI, 
WebOOGL  from  the  Geometry  Center,  AmberGL  from  DIVE  Lab.,  WorldView  for 
Intervista  Software,  Inc.  With  these  browsers,  you  can  view  and  walk  through  a 
virtual  world  described  by  the  retrieved  VRML  file.  However,  none  of  these 
browsers  has  fulfilled  the  original  goal  of  multiple-user  interactivity,  even  for  those 
browsers  that  have  implemented  VRML  version  2.0  released  in  August  1996. 

In  this  report,  the  author  will  describe  the  design  and  implementation  of 
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SharetlWeb,  a  web  browsing  system  that  can  support  the  interaction  among  clients 
over  the  existent  WWW  environment. 

2.  Multi-user  Browser  System 

Utilizing  the  existing  WWW  environment  to  build  a  multi-user  interactive 
system,  new  communication  protocols  need  to  be  added  to  provide  the  interactions 
among  distributed  web  browsers.  Instead  of  modifying  the  existing  HTTP  protocols 
WWW  is  using,  the  goal  is  to  design  a  new  multi-user  virtual  system  and  seamlessly 
integrate  it  with  the  WWW  architecture.  For  a  distributed  multi-user  system  to  be 
seamlessly  integrated  with  the  WWW  environment,  it  must  have  the  following 
characteristics:  [3] 

1 .  The  user  can  download  a  scene  tile  from  any  WWW  server  with  the  HyperText 
Transfer  Protocol  (HTTP). 

2.  The  multi-user  system  must  provide  the  hyperlink  feature  to  retrieve  various  media 
resources  that  are  supported  by  the  WWW  environment. 

3.  The  multi-user  system  must  be  able  to  handle  information  exchange  between 
HTML  documents  and  the  objects  inside  a  virtual  world. 

4.  The  multi-user  system  server  itself  is  an  add-on  of  an  existing  WWW  server. 

5. 1  he  user  can  directly  enter  a  virtual  world  through  the  WWW  server  from  which 
the  scene  file  is  downloaded. 

In  summary,  the  seamless  integration  implies  that  the  user  can  simply  download  a 
virtual  scene  file  from  the  supporting  WWW  server,  navigate  through  the  virtual 
worlds  managed  by  the  WWW  servers,  and  fully  utilize  the  services  provided  by  the 
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WWW  environment  without  the  user’s  awareness. 

The  researches  on  the  multi-user  virtual  environment  can  be  classified  into  two 
categories:  network-based  and  web-based  virtual  reality  systems.  The  network-based 
system  researchers  generally  focus  on  providing  a  large-scale  environment  with  every 
available  network  techniques.  On  the  other  hand,  the  web-base  system  researchers 
emphasize  on  providing  multi-user  virtual  environment  on  top  of  the  existent  WWW 
environment.  Although  our  work  is  concentrated  on  designing  a  multi-user  virtual 
reality  system  over  the  existent  WWW  environment,  the  techniques  from  the 
network-base  multi-user  system  also  provide  partial  solutions  to  our  work. 

The  WWW  environment  is  a  client/server  architecture  using  HTTP  protocol. 

Due  to  the  intrinsic  properties  of  HTTP,  the  WWW  system  is  a  one  way  street  that 
can  not  achieve  interactivity  among  users.  Hence,  new  mechanisms  and  protocols 
have  to  be  designed  to  support  the  multi-user  interaction  over  the  WWW 
environment.  The  VRML  standard  is  an  attempt  to  deliver  the  virtual  world  on  the 
WWW  environment,  and  its  original  goal  is  to  support  the  interaction  of  multiple 
participants.  However,  even  the  VRML  version  2.0  [2]  failed  to  define  the  standard 
of  the  multiple  user  interactions.  Therefore,  several  designers  of  the  web-based 
multi-user  systems  have  developed  their  own  proprietary  methods  to  support  the 
multi-user  interaction.  [4—5] 

Virtual  Society  (VS)  is  a  research  project  of  Sony  Computer  Science  Laboratory, 
Inc.  in  attempt  to  define  a  global  architecture  and  a  set  ot  protocols  to  realize  a  multi¬ 
user  interactive  3D  environment  in  a  WWW  setting.  [4]  It  was  originally  based  on 
the  DIVE  platform  developed  by  the  Swedish  Institute  of  Computer  Science.  [6]  The 
VS  project  has  developed  a  VRML-standard  virtual  reality  system  over  the  WWW 
environment,  called  Community  Place  Browser.  [7]  The  VS  architecture,  however,  is 
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not  seamlessly  integrated  with  the  WWW  environment.  For  example,  the  VS  client 
can  not  enter  a  virtual  world  from  the  WWW  server  from  which  the  scene  file  was 
downloaded.  After  a  scene  file  is  downloaded  from  a  WWW  server,  the  VS  client 
must  send  a  URL  to  that  WWW  server  to  query  the  necessary  information,  such  as  IP 
address  and  port  number,  of  the  VS  server.  In  addition,  the  VS  server  does  not 
support  information  exchange  between  the  HTML  document  and  objects  inside  a 
virtual  world.  Hence,  VS  is  an  extension  of  the  networked  virtual  reality  system  to 
the  WWW  environment  rather  than  a  merge  of  the  two. 

The  German  National  Research  Center  for  Information  Technology  (GMD)  has 
proposed  another  type  of  web-based  multi-user  system  by  defining  new  nodes  for 
VRML  standard.  [5]  Different  from  the  VS  project,  the  multi-user  system  by  GMD 
does  not  require  a  multi-user  server  to  maintain  the  consistency  among  distributed 
objects.  To  achieve  the  consistent  control  of  shared  objects,  GMD  proposed  a  so- 
called  Active  Lock  mechanism  and  two  extension  nodes,  Interaction  node  and 
Behavior  node,  for  the  VRML  standard.  The  research  of  GMD  has  been 
concentrating  on  extending  the  VRML  standard  for  the  multi-user  interaction.  Issues 
of  seamless  integration  discussed  in  the  previous  section  still  require  further  study. 

Scalable  Platform  for  Large  Interactive  Networked  Environment  (Spline)  by 
Mitsubishi  Electric  Research  Lab.  (MERL)  uses  yet  another  approach  to  integrate  the 
multi-user  virtual  environment  with  the  WWW  environment.  [8]  Spline  is  essentially 
a  networked-based  multi-user  virtual  reality  system  that  uses  Interactive  Sharing 
Transfer  Protocol  (ISTP)  as  its  communication  protocol.  [9]  ISTP  incorporated 
several  simple  HTTP  protocols  to  enable  the  HTTP  server  to  participate  in  multi-user 
interactions.  With  the  help  of  ISTP,  the  Spline  system  provides  a  WWW-like  virtual 
environment.  However,  it  is  not  a  web-based  multi-user  system. 
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Other  web-based  multi-user  systems  that  are  commercially  available  include 
Ccpro  [10]  from  Blaxxun  interactive  Corp.,  OnLive!  Community  browser  [11]  from 
OnLive!  Technologies  Inc.,  OZ  Virtual  [12]  from  OZ  Interactive  Inc.  and  V*Realm 
Multi-user  browser  [  1 3]  from  Integrated  Data  Systems,  Inc.  All  of  the  above 
browsers  adopt  the  VRML  standard  as  their  virtual  scene  file  format.  Each  of  the 
above  multi-user  systems  uses  its  own  technique  to  provide  interactivity  among 
distributed  users.  None  of  the  above  browsers  is  fully  integrated  with  the  traditional 
HTML  browser.  Active  World  browser  [14]  from  Worlds  Inc.  is  another  kind  of 
multi-user  browser  that  is  not  a  VRML  browser,  whereas  it  merges  the  traditional 
HTML  browser  into  its  own  browser.  However,  it  can  not  handle  information 
exchange  between  HTML  documents  and  the  objects  inside  a  virtual  world. 

3.  Design  Considerations 

Since  the  existing  WWW  browsers  have  difficulties  in  creating  an  interactive 
multi-user  virtual  world,  the  SharedWeb  system,  a  three-dimensional  web  browsing 
system,  is  designed  to  support  the  interactions  among  the  clients  over  the  WWW 
environment.  [15] 

3.1  Mechanisms  to  Support  the  Multi-user  Interactivity  over  WWW 

There  are  four  intrinsic  problems  must  be  solved  to  support  the  multi-user 
interaction  over  the  WWW  environment.  They  are  Client-Information  Recording, 
Server-to-Client  Callback,  Excessive  Network  Load,  and  Virtual  World  Entrance. 


3.1.1  Client-Information  Recording 

The  WWW  server  uses  a  request-and-response  technique  to  serve  its  client’s 
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request.  The  link  between  the  server  and  client  is  established  only  when  a  client 
issues  a  request  to  the  server.  After  the  requested  information  is  sent,  the  link  is 
broken  and  the  client  is  “forgotten”  by  the  server  immediately.  However,  in  a  multi¬ 
user  interaction  system,  the  server  needs  to  keep  track  of  all  the  users.  Since  the  link 
is  broken  immediately  after  the  requested  information  is  sent  to  the  client,  the  WWW 
server  does  not  keep  the  records  of  the  clients  who  have  requested  information.  We 
call  this  the  Client  Information  Recording  problem. 

To  solve  this  problem,  each  browser  must  be  able  to  provide  a  sequence  number, 
called  Unique  Identification  (UID),  to  uniquely  identify  Itself  to  the  server.  Modified 
from  Chin's  proposal  [16-17],  the  SharedWeb  browser  uses  the  combination  of  Login 
name,  Time  stamp  and  listen  Port  number  (LTP)  as  the  UID  to  identify  itself.  The 
login  name  is  the  nickname  of  the  user  to  login  to  a  virtual  world,  the  value  of  the 
time  stamp  is  the  time  when  the  browser  was  first  activated  and  the  listen  port  number 
is  the  port  number  that  the  browser  used  to  receive  messages.  With  this  UID  number, 
the  SharedWeb  server  can  easily  identify  a  new  registered  user  and  assigns  a  User  ID 
to  that  browser.  Moreover,  the  SharedWeb  server  will  record  this  UID  number  for 
future  reference. 

3.1.2  Server-to-Client  Callback 

The  link  between  the  server  and  the  client  is  established  only  when  a  client 
issues  a  specific  request.  The  WWW  server  does  not  actively  set  up  a  connection  to 
its  clients;  therefore,  it  does  not  have  the  ability  to  forward  information  from  one 
client  to  other  clients.  This  is  referred  as  the  Server-toCIient  Callback  problem. 

To  solve  this  problem,  the  SharedWeb  server  is  dedgned  to  have  the  ability  of 

actively  set  up  connection  the  clients  and  forwarding  information  from  one  user  to 
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other  users. 


Another  technique  being  widely  discussed  recently  is  “Cookies”.  [18-19]  The 
Cookie  technique  is  based  on  a  two-stage  process.  First,  a  “cookie”,  or  a  small  piece 
of  information,  is  sent  by  the  Web  server  and  stored  in  the  user’s  computer  without  its 
consent  or  knowledge.  Second,  the  cookie  clandestinely  and  automatically 
establishes  connection  to  the  Web  server.  In  its  current  form,  Cookies  can  not  be  used 
as  the  Callback  mechanism  from  WWW  server. 

3.1.3  Excessive  Network  Load 

Since  the  link  between  WWW  server  and  client  is  established  only  when  a 
request  is  issued,  in  addition  to  the  intrinsic  network  latency,  there  is  a  delay 
associated  with  establishing  the  link.  A  central  server  is  usually  used  in  a  wetnbased 
multi-user  virtual  system  to  mediate  the  communication  among  the  clients.  [5]  The 
network  bandwidth  will  be  overloaded  very  quickly  as  the  number  of  users  increases, 
and  the  server  will  rapidly  become  an  input/output  bottleneck.  We  describe  this 
situation  as  an  Excessive  Network  Lead  problem. 

Distributed  Interactive  Simulation  (DIS)  [20]  is  a  technique  developed  by  the 
Department  of  Defense  for  military  training.  It  is  a  fully  distributed  simulation 
environment  with  no  centralized  control.  In  DIS,  a  concept  called  Dead  Reckoning  is 
employed  to  reduce  the  network  traffic  problem.  [21-22]  The  Dead  Reckoning  model 
uses  the  current  position  and  velocitv  acceleration  to  predict  the  future  positions.  All 
entities  use  the  same  dead  reckoning  model  to  update  the  position  of  one  particular 
entity  to  achieve  consistency.  Each  entity  is  responsible  for  comparing  its  own  dead 
reckoning  model  with  the  true  model  When  the  error  exceeds  a  predefined  threshold 
value,  it  needs  to  inform  all  entities  its  new  position.  SharedWeb  can  adopt  the  same 
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concept.  The  states  of  all  avatars  are  updated  based  on  the  respective  dead  reckoning 
models  until  the  new  status  change  information  of  one  particular  avatar  is  received. 
The  information  is  sent  in  the  form  of  a  self-contained  data  package,  called  Protocol 
Data  Unit  (PDU),  and  forwarded  to  each  participating  browser  through  the 
SharedWeb  server.  The  PDU  is  sent  to  the  SharedWeb  server  only  when  the  deviation 
of  the  dead  reckoning  model  from  the  true  model  exceeds  a  predefined  threshold. 
Hence,  the  bandwidth  requirement  can  be  reduced. 

3.1.4  Virtual  World  Entrance 

For  a  communication  protocol  to  support  the  multiuser  interaction  over  the 
WWW  environment,  the  seamless  integration  is  important  for  an  adopted  virtual 
environment  to  take  full  advantage  of  the  WWW  environment.  The  user  has  to  be 
able  to  set  up  connection  to  the  server  without  hassle.  The  problem  is  referred  as 
Virtual  World  Entrance. 

To  solve  this  problem,  we  can  either  embed  the  multi-user  server  information 
into  the  downloaded  scene  file  or  allow  the  client  to  issue  a  query  message  to  the 
WWW  server  for  the  information  of  the  multi-user  server.  Although  the  first  option 
is  simpler,  this  method  will  cause  trouble  when  a  multi-user  server  is  crashed  and  a 
replacement  server  is  activated.  As  to  the  second  option,  the  client  has  to  wait  until 
the  replied  HTML  document  is  received  before  it  can  be  connected  to  the  virtual 
world  as  Virtual  Society  from  Sony  SCL  did.  [23]  Both  methods  fail  to  satisfy  the 
criteria  of  the  seamless  integration  as  the  outline  in  Section  2. 

The  WWW  server  has  to  treat  the  multi-user  server  as  an  add-on  application  in 
order  to  integrate  the  network  virtual  system  into  the  WWW  environment  seamlessly. 
In  the  WWW  architecture,  the  WWW  server  uses  the  Common  Gateway  Interface 
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(CGI)  to  communicate  to  its  add-on  application.  [24]  Hence,  a  CGI  program  can  be 
implemented  to  forward  client  information  from  the  WWW  server  to  the  multi-user 
server.  Using  this  approach,  whether  the  multi-user  server  and  the  WWW  server 
reside  in  the  same  host  does  not  matter  to  the  client.  In  addition,  a  hierarchy  of  the 
multi-user  servers  can  be  accordingly  implemented.  The  WWW  server  passes  the 
login  information  of  a  client  to  a  load-monitor  program,  which  will  in  turn  forward 
the  client  to  one  of  the  multi-user  servers  without  the  awareness  of  the  client.  In  this 
case,  the  multi-user  server  system  can  have  a  better  load  balance  and  fault  tolerance. 


3.2  Time  Synchronization 

Time  synchronization  is  another  important  issue  in  the  multi-user  virtual  system. 
The  multi-user  virtual  environment  is  a  synthetic  world  on  the  network  where  users 
can  navigate  and  interact  with  each  other.  In  our  real  world,  all  people  use  the  same 
standard  time  to  synchronize  their  local  clocks  and  interact  with  each  other.  In  the 
virtual  world,  however,  the  users  join  the  scene  from  different  computers,  and  the 
time  in  each  user’s  virtual  world  is  decided  by  the  respective  computer’s  system  time. 
The  time  needs  to  be  concordant  for  all  users  in  the  same  virtual  world  so  that  the 
event  actions  are  referring  to  the  same  clock.  It  is  the  Time  Synchronization  problem. 

The  Network  Time  Protocol  (NTP)  has  been  used  to  synchronize  the  time  of  a 
computer  client  or  server  to  another  server  or  reference  time  source,  such  as  a  radio  or 
satellite  receiver  or  modem.  It  provides  client  accuracy  typically  within  a  millisecond 
on  LANs  and  up  to  a  few  tens  of  milliseconds  on  WANs  relative  to  a  primary  server 
synchronized  to  Coordinated  Universal  Time  (UTC)  via  a  Global  Positioning  Service 
(GPS)  receiver,  for  example.  Typical  NTP  configurations  utilize  multiple  redundant 
servers  and  diverse  network  paths,  in  order  to  achieve  high  accuracy  and  reliability. 
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Some  configurations  include  cryptographic  authentication  to  prevent  accidental  or 
malicious  protocol  attacks.  [25] 

The  NTP  architecture,  protocol  and  algorithms  have  evolved  over  almost  two 
decades,  with  the  latest  NTP  Version  3  and  4  designated  an  Internet  standard.  Among 
the  goals  of  this  design  are:  [26] 

1 .  Optimize  the  computer  clock  accuracy  and  stability,  subject  to  constraints  of 
network  overheads  and/or  telephone  toll  charges,  relative  to  local  and/or  remote 
sources  of  time. 

2.  Enhance  the  reliability  by  detecting  and  discarding  misbehaving  local  and/or 
remote  sources  and  reconfiguring  network  paths  as  necessary. 

3.  Automatically  adjust  algorithm  parameters  in  response  to  prevailing  network 
delay/jitter  conditions  and  the  measured  stability  of  the  computer  clock  oscillator. 

The  NTP  technique  can  help  the  SharedWeb  system  to  achieve  time 
synchronization.  All  users  synchronize  their  time  with  SharedWeb  server  using  the 
NTP  technique  before  they  enter  the  global  virtual  world.  It  can  ensure  that  one  event 
happens  at  the  same  time  for  all  users  in  the  same  virtual  world. 


4.  The  SharedWeb  System 

With  the  mechanisms  presented  in  the  previous  section,  the  SharedWeb  system 
provides  a  natural  extension  of  the  WWW  environment  to  a  multi-user  virtual 
environment.  The  infrastructure  of  the  SharedWeb  system  is  shown  in  Figure  1.  The 
nodes  of  the  SharedWeb  system  can  be  classified  into  two  types:  the  server  site  and 
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the  browser  site.  The  browser  site  is  composed  of  four  modules:  Multiple 
Participants  Interface,  3D  Render  Engine,  Chat  Phase,  and  WWW  Homepage  Viewer 
modules.  The  server  site  is  built  on  top  of  the  existent  WWW  server  with  two 
modules:  a  CGI  program  and  the  SharedWeb  server.  The  server  and  the  browser  are 
communicated  by  the  SharedWeb  Communication  Protocol  ^SWCP),  which  will  be 
fully  explored  in  the  next  subsection.  Each  of  these  modules  is  responsible  for  the 
tasks  described  as  below: 


Figure  1.  Infrastructure  of  the  SharedWeb  system. 


4.1  The  Browser  Site 

The  Multiple  Participant  Interface  module  runs  on  tor  of  Winsock  to  manipulate 
the  interaction  between  the  browser  and  other  users.  This  module  is  the  kernel  of  a 
SharedWeb  browser.  The  main  functions  of  this  module  are  to  calculate  browser's 
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UID,  encapsulate  and  de-capsulate  the  PDUs,  transmit/receive  data  to/from  the 
network,  and  coordinate  messages  among  3D  Render  Engine,  Chat  Phase,  and  WWW 
Homepage  Viewer  modules. 

The  3D  Render  Engine  module  provides  a  visual  window  for  the  user  to  explore 
the  virtual  world.  When  a  browser  retrieves  a  scene  file  from  the  WWW  server,  it  is 
an  ASCII  file.  The  scene  file  will  then  be  translated  into  a  virtual  world  database 
recognized  by  the  3D  Render  Engine  module. 

The  Chat  Phase  module  is  designed  to  support  text  communication  among 
distributed  users.  This  module  enhances  the  interactions  among  distributed 
participants  and,  hence,  broadens  the  applicability  of  the  SharedWeb  system.  Without 
this  module,  the  shared  virtual  world  will  be  a  wordless  world  that  makes  the  user 
bored  easily  when  playing  with  the  SharedWeb  browser. 

The  WWW  Homepage  Viewer  module  is  a  standard  HTML  browser.  By 
seamlessly  integrating  this  module  into  the  SharedWeb  browser,  a  user  can  enter 
another  virtual  world  by  double  clicking  a  hyperlink  provided  on  a  Homepage  or 
inside  the  virtual  world.  Further,  the  user  can  exchange  information  with  each  other 
by  using  the  Homepage  and  the  HTTP  protocol. 


4.2  The  Server  Site 

The  WWW  server  is  a  gateway  for  a  SharedWeb  user  to  enter  a  virtual  world. 

For  the  user,  he  only  needs  to  download  a  scene  file  from  this  WWW  server  and  to 
losin  his  nickname  to  enter  a  virtual  world.  After  the  WWW  server  receives  the  login 
information  from  a  browser,  the  WWW  server  will  pass  the  received  user  information 
to  the  SharedW'eb  server  through  the  CGI  program  mentioned  in  the  previous  section. 

The  SharedWeb  server  module  is  the  kernel  of  the  entire  SharedWeb  system. 
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This  program  is  responsible  for  mediating  the  information  exchange  among  the 
browsers  in  the  same  virtual  world.  That  is,  all  the  client  information  is  sent  to  the 
server  and  the  server  is  responsible  for  routing  the  received  information  to  the 
browsers  in  the  same  virtual  world,  with  the  exception  of  the  chat  (see  Chat  PDU 
below)  among  users. 

4,3  SharedWeb  Communication  Protocol  (SWCP) 

There  is  no  standardized  communication  protocol  for  the  web-based  multi-user  virtual 
system.  Different  protocols  have  been  proposed  by  various  systems  over  the  years. 
[4~5][9]  The  communication  protocol  includes  steps  to  join  a  virtual  world,  to 
forward  messages  among  distributed  users,  and  to  leave  the  virtual  world.  On  the 
other  hand,  the  DIS  standard  is  by  far  the  most  successful  technique  that  enables  a 
fully  distributed  virtual  environment  to  be  built  on  a  network.  The  DIS  standard  [20] 
defines  the  data  formal.  PDU,  and  the  communication  protocol  that  employs  various 
PDUs  to  achieve  the  spatial  consistency  among  distributed  users. 

Although  the  DIS  standard  has  been  proven  to  be  an  effective  method  to  support 
the  networked  multi-user  virtual  environment  on  the  Internet,  it  is  not  designed  for  the 
web-based  multi-user  \irtual  environment.  Inspired  by  the  DIS  technique,  the 
SharedWeb  system  adopts  the  concept  of  PDUs  to  provide  the  multi-user  interaction 
over  the  WWW  environment.  The  PDU  design  focuses  fully  on  the  integration  of  the 
virtual  system  into  the  WWW  architecture.  Seven  PDUs  are  defined  in  the 
SharedWeb  system:  [27] 

1.  Initial  PDU:  This  PDU  is  used  to  submit  the  information  required  to  log  into  a 
virtual  world,  and  to  inform  other  users  that  a  new  user  just  joined. 
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2.  Acknowledge  PDU:  When  the  SharedWeb  server  receives  an  Initial  PDU  from  a 
browser,  it  will  respond  with  an  Acknowledge  PDU  to  the  browser. 

3.  End  PDU:  When  a  user  wants  to  exit  from  the  virtual  world,  it  will  send  an  End 
PDU  to  notify  the  SharedWeb  server  and  all  other  users. 

4.  Object  State  PDU:  Browser  uses  this  PDU  to  send  the  status-change  information  to 
the  SharedWeb  server  and  other  browsers.  To  reduce  network  traffic,  this  PDU  is 
issued  only  when  the  dead-reckoned  position  of  the  avatar  deviates  from  its  actual 
position  by  a  predefined  threshold  value. 

5.  Chat  PDU:  SharedWeb  system  supports  the  peer-to-peer  chatting  protocol.  This  is 
the  only  PDU  that  bypasses  the  SharedWeb  server.  It  can  prevent  the  bottleneck  on 
the  SharedWeb  server  site  as  the  number  of  user  increases. 

6.  HTTP  PDU:  This  PDU  his  two  functions  in  this  system.  First,  this  PDU  allows  the 
user  to  send  a  URL  address  of  a  network  resource,  such  as  an  HTML  document  or 
a  audio  clip,  to  other  users.  Second,  it  lets  the  users  control  the  objects  in  the 
virtual  world  from  an  HTML  document.  Whenever  data  are  filled  into  an  HTML 
document  provided  by  the  SharedWeb  browser,  this  information  is  sent  to  its  own 
Web  server,  as  shown  in  Figure  1.  The  CGI  decodes  the  document  and  forward  the 
information  to  the  SharedWeb  server.  The  SharedWeb  server  then  packs  the 
information  into  an  HTTP  PDU  and  forwards  to  other  browsers. 

7.  Error  PDU:  When  an  error  occurs  between  the  clients  and  server,  users  involved 
are  informed  using  this  PDU. 

4.4  User  Interaction  with  the  SharedWeb  System 

The  user  interactions  ir.  the  SharedWeb  system  include  six  steps:  Initialization, 

Login,  Update,  Chat,  HTTP,  and  End.  [28]  The  SharedWeb  browser’s  start-up  page  is 
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shown  in  Figure  2. 
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Figure  2.  Start-up  page  of  the  SharedWeb  browser. 


4.4.1  Initialization  step 

This  i>  the  first  step  a  user  joins  the  multi-user  system.  Once  launched,  the 
SharedWeb  browser  (client)  shows  a  homepage  with  a  list  of  exercises  available,  as 
shown  in  the  left  window  in  Figure  2.  The  user  selects  the  exercise  and  downloads 
the  virtual  world  scene  file  and  the  related  information  document. 


4.4.2  Login  step 

The  i:>er  keys  in  a  nickname  of  his/her  choice  in  the  lower-right  “Input  User 
Name”  window  in  Figure  2.  After  the  SharedWeb  server  receives  the  Initial  PDU,  it 
will  register  the  user  and  send  an  Acknowledge  PDU  back  to  the  SharedWeb  browser. 
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Figure  3  is  an  example  of  the  browser  after  the  registration  is  completed. 


Figure  3.  Viewpoint  of  the  red  tank  commander  after  registration. 

4.4.3  Update  step 

Tr.c  user  now  can  start  to  navigate  the  avatar  in  the  virtual  world.  The 
ShareuWeb  browser  provides  a  control  bar  and  an  HTML  form,  as  shewn  in  Figure  3. 
The  user  can  use  either  one  to  control  the  avatar.  When  the  dead-reek  :ned  position  of 
T  •  the  avatar  deviates  from  its  actual  position  by  a  predefined  threshold  \ulue,  the 

browser  automatically  sends  an  Object  State  PDU  to  the  server  to  inform  other  users 
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in  the  same  virtual  world  its  update  position. 
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4.4.4  Chat  step 
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Tire  user  uses  a  chat  window,  as  shown  in  Figure  3.  to  send  text  messages  to  any 
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or  all  users  in  the  same  virtual  world.  The  SharedWeb  has  adopted  the  Internet  Relay 
Protocol  [29]  in  its  Chat  PDU  design. 

4.4.5  HTTP  step 

The  SharedWeb  system  can  integrate  Web  resources  into  the  virtual  world.  This 
is  achieved  by  implementing  the  HTTP  PDU.  The  user  can  send  a  URL  address  to 
other  users  in  the  same  virtual  world.  The  SharedWeb  browser  receiving  the  HTTP 
PDU  will  auto-download  this  HTML  document  from  the  addressed  WWW  server. 

4.4.6  End  step 

The  user  can  exit  from  the  virtual  world  by  select  “Exit  World”  under  the  “File” 
menu.  The  browser  sends  an  End  PDU  to  the  SharedWeb  server. 

5.  SharedWeb  System  Implementation 

The  SharedWeb  system  is  implemented  using  Microsoft  Visual  C++  version  5.0. 
The  SharedWeb  server  runs  on  Windows  NT  Server  version  4.0  with  an  embedded 
WWW  server.  The  SharedWeb  browser  is  currently  available  as  a  standalone 
browser.  In  the  future,  it  will  be  implemented  as  a  plug-in  application  for  the 
Netscape  Communicator  and  Internet  Explorer.  The  user  interface  of  the  SharedWeb 
browser  is  shown  in  Figure  2  and  3. 

As  displayed  on  the  upper-right  window  of  Figure  2,  Viscape  from  Superscape 
Inc.  [30]  is  used  as  the  3D  Render  Engine.  When  the  browser  is  initiated,  the  user 
will  get  SharedWeb  system  information  in  the  left  HTML  window.  After  the  user  has 
successfully  logged  in  a  virtual  world,  the  Chat  Input  Box  will  be  activated 
automatically  and  all  of  the  users’  names  inside  the  virtual  world  will  be  listed  in  the 
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User  Lisi  Window.  Further  more,  a  "Send  URL”  button  will  emerge  at  the  same  time, 
as  shown  in  Figure  4.  The  “Send  URL”  button  is  a  special  function  for  the  user  to 
send  a  hyperlink  resource,  such  as  an  HTML  document  or  audio  clip,  to  other  users. 
This  function  is  another  feature  for  the  SharedWeb  system  to  be  seamlessly  integrated 
into  the  WWW  environment. 


LafcTamKahg^ShaiedWeb : 


File  -View  Help'*'"!--; . 


■  Back 

rr 

Home 

Reload 

:Stop 

v*  y ; *vv  *  '  ■*  -  $ 

URL  h»p:A*V5tan5.isLucf.eduA^carnpi.is(e).hr:.t  -▼[ 


Virtual  Campus 


JW  -v-:A: 

«3S2SSSSXS£2S232SSS3SSCSS3K 


Virtual  campus  is  a 
full-scale  reconstruction  of 
the  campus  of  Tamkaitg 
University,  Tainsui,  Taipei, 
Taiwan,  with  the  virtual 
reality  technique.  This 
application  is  a 


.  "M 

- 


-•-1 


Oil 

*/ 

V  <  *. 

\'t  •  d  Mill 

- . - . — .  -  i 

n  r-.? 

Jim  :  Hi  Lin! 

Jim  :  How  are  you,  Lin?_| 


Uirfiood!! 


Jim 

Tom 


pJ 

'  Send  URL  ' 

i.i 

1 

Chat: 

■'  '  - 

Figure  4.  A  scene  in  the  virtual  campus. 


After  rue  "Semi  URL"  button  is  clicked,  it  v.  ill  disappear  and  a  highlighted 
message  u.ll  be  uLpiayed  in  Chat  Output  Window  to  notify  the  user.  In  the 
meantime,  the  label  of  the  Chat  Input  Box  is  changed  to  “URL:”  for  the  user  to  input 
the  URL  address  of  a  network  resource,  such  as  art  HTML  document  or  aueio  clip. 
Moreover,  the  sender  selects  the  receiver(s)  by  clicking  one  or  more  nameis)  from  the 
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User  List  Window.  By  clicking  the  OK  button,  the  specified  network  resource  will 
then  be  automatically  forwarded  to  the  receiver(s). 

To  fully  explore  the  features  provided  by  the  SharedWeb  system,  four  virtual 
worlds  have  been  constructed.  The  following  scenarios  are  the  virtual  worlds 
currently  supported  in  the  SharedWeb  environment.  The  user  can  move  from  one 
scenario  to  another  without  logging  out. 

5.1  Virtual  Office 

Virtual  Office  is  an  application  of  the  SharedWeb  system  to  demonstrate  the 
benefits  of  integrating  an  HTML  browser  with  a  3D  browser.  The  scene  of  this 
virtual  world  is  shown  in  Figure  5.  The  scenario  of  this  demonstration  is  the  Office 
of  Administration.  The  user  can  request  an  application  form  in  the  HTML  format. 
The  form  requested  will  appear  on  the  left  window  in  Figure  5.  After  filling  the  form, 
the  user  can  submit  it  by  clicking  the  “Confirm"  button.  The  chatting  capability  in 
this  demonstration  provides  a  means  for  the  student  to  ask  questions  in  case  he/she 
has  any  questions  about  the  procedures. 
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Figure  5.  A  scene  in  the  Virtual  Office  and  the  form  requested  in  HTML  form. 


Three  methods  are  provided  by  the  Shared  Web  system  for  students  to  access  the 
application  form.  The  first  method  is  to  ask  a  virtual  employee,  who  also  logs  in  via  a 
SharedWeb  Browser,  in  the  virtual  office  to  send  him/her  the  application  form.  The 
second  wav  to  retrieve  the  application  form  is  to  c.ick  a  virtual  photo  on  the  wall  that 
represents  ;.  particular  application  form.  The  third  way  for  the  student  to  access  any 
application  form  is  to  click  the  HTML  hyperlinks. 
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5.2  Virtual  Campus 

The  v  irtual  campus  is  modeled  with  the  campus  of  Tamkang  University  in 
Taiwan.  The  user  can  vv  a.k  through  the  campus  ar.d  access  to  different  virtual 
buildings  and  virtual  offices.  Figure  4  in  Section  5  shows  an  example  of  the  scenes 
the  user  sees  in  the  virtual  campus.  The  Office  of  Administration  in  scenario  5.1  is 
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one  of  the  virtual  offices  the  user  can  enter. 


5.3  Bumper  Cars 

This  is  a  game  of  bumper  cars.  Each  user  controls  his/her  own  car  and  bumps 
with  other  cars,  as  shown  in  Figure  6. 


Figure  6.  A  scene  in  the  Bumper  Cars. 


5.4  Tank  Commander 

This  interactive  tank  corneal  game  is  a  prototype  a  system  integrating  the  3D 
technique  and  WWW  environ.uent.  This  application  is  a  3D  version  of  the  role- 
playing  game  tRPGj  simulation. 

Traditionally,  RPG  is  a  two-dimensional  graphical  display  environment.  It  has 
an  interactive  dialog  box,  or  command  sheet,  tor  the  user  to  control  his  character 
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inside  the  game.  Since  the  SharedWeb  system  fully  integrates  the  HTML  browser 
with  the  virtual  reality  navigator,  the  command  sheet  can  be  easily  implemented  by 
the  HTML  document  so  that  the  content  of  the  command  sheet  can  be  dynamically 
modified.  Also,  a  game  master  is  required  for  RPG  to  control  the  game  rules.  For  the 
Shared  Web  system,  the  game  master  was  implemented  in  the  SharedWeb  server. 

The  prototype  system  allows  three  users  to  enter  virtual  battlefield,  the 
‘‘bluetank,”  the  “redtank,”  and  the  “referee”.  Bluetank  and  redtank  can  use  the 
HTML  document  to  control  two  tanks  each  in  the  virtual  battlefield.  Figure  3  in 
Section  4.4.2  shows  the  viewpoint  of  the  red  tank  commander.  The  user  command 
sheet  has  automatically  downloaded  into  the  HTML  browser.  The  user  can  input  data 
to  control  tank  movements,  la  addition,  check  buttons  allow  the  user  to  choose 
whether  or  not  to  fire  tank  missiles.  After  the  user  inputs  the  commands  and  presses 
the  “Confirm”  button,  an  HTTP  PDU  is  sent  to  the  Sharedweb  server.  The  server 
then  forwards  it  to  all  participants  and  the  status  of  the  tank  is  modified  accordingly. 

6.  Conclusion  and  Future  Works 

The  SharedWeb  system  has  been  successfully  tested  in  both  the  Local  Area 
Network  (LAN)  and  Wide  Area  Network  (WAN)  environment.  The  major  concern  is 
the  time  delay  in  the  information  exchange  among  the  SharedWeb  browsers.  The 
time  delays  in  the  LAN  environment  are  very  small.  In  WAN,  however,  the  network 
congestion  is  always  a  concern.  The  SharedWeb  system  uses  the  Dead  Reckoning 
technique  to  reduce  the  network  traffic  created  by  the  interactions  among  the  users. 
However,  there  is  not  much  can  be  done  to  deal  with  the  congestion  in  the  Internet 
itself.  After  all,  unlike  DIS-which  is  running  in  an  isolated  network—  all  the  users  in 
the  world  share  the  Internet.  We  have  no  control  over  when  other  users  would  create 
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a  heavy  traffic  in  the  Internet.  In  any  case,  the  Dead  Reckoning  technique  can 
maintain  the  interactivity  even  when  the  congested  network  delays  the  PDUs.  After 
receiving  a  delayed  PDU.  proper  delay  compensation  and  smoothing  are  needed  to 
make  the  interaction  realistic.  [31] 

This  report  has  presented  the  concept  of  seamless  integration  of  the  multi-user 
system  into  the  WWW  environment.  With  this  seamless  integration,  the  existing 
WWW  server  can  easily  support  multi-user  interactivity,  and  a  shared  virtual  world 
can  thus  be  easily  constructed.  The  infrastructure  and  the  communication  protocol  of 
this  concept  have  been  fully  elaborated  in  this  report.  In  addition,  the  architecture  of 
a  web-based  multi-user  system,  called  the  SharedWeb  system,  has  also  been 
presented.  Under  the  architecture  of  the  SharedWeb  system,  the  existing  WWW 
server  remains  intact  while  supporting  the  multiple  participant  interactions.  More 
significantly,  the  SharedWeb  environment  is  transparent  to  the  users.  That  is,  each 
user  only  needs  to  download  a  scene  file  from  a  WWW  server  and  logins  with  his 
nickname,  and  the  WWW  server  will  then  gracefully  transfer  the  user  to  an 
appropriate  SharedWeb  server. 

Although  the  SharedWeb  system  successfully  demonstrates  the  concept  of 
seamless  integration,  more  researches  are  still  required  to  improve  the  multi-user 
interaction.  For  example,  the  consistency  control  of  the  shared  objects  is  an  important 
function  for  computer-supported  cooperative  work  (CSCW)  type  of  applications.  [32] 
Moreover,  in  a  virtual  environment  with  N  avatars,  the  current  SharedWeb  server 
must  “broadcast”  a  change  in  any  avatar’s  state  to  N  -  1  browsers.  Thus,  in  the  worst 
case,  the  server  must  send  -V 2  -  A'  PDUs  to  the  network.  This  approach  yields  0(N 
2)  complexity  for  updating  messages  and  could  exhaust  the  computation  power  of  a 
host  even  before  it  saturates  the  network.  To  overcome  this  problem,  the  concept  of 
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spatial  culling  [33]  is  currently  under  investigation  for  the  SharedWeb  system. 

Multi-server  architecture  of  SharedWeb  system  is  another  important  future  work. 
This  architecture  can  further  scale  up  the  number  of  participants  as  well  as  provide 
the  fault  tolerance  feature  on  the  server  site. 
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Abstract 


The  classical  problem  of  optimum  detection  of  a  complex  signal  of  unknown  amplitude  in 
colored  Gaussian  noise  is  revisited.  The  focus,  however,  is  on  adaptive  system  designs  with 
limited  training  data  sets  and  low  computational  optimization  complexity.  In  this  context,  the 
target  vector  is  equipped  with  a  carefully  selected  orthogonal  auxiliary  vector  for  disturbance 
suppression  with  one  complex  space-time  degree  of  freedom.  Direct  generalization  leads  to 
adaptive  generation  of  a  sequence  of  conditionally  optimized  weighted  auxiliary  vectors  that 
are  orthogonal  to  each  other  and  to  the  target  vector  of  interest.  This  approach  appears  here  for 
the  first  time.  Adaptive  disturbance  suppression  with  any  desired  number  of  complex  degrees 
of  freedom  below  the  data  dimension  is  therefore  possible.  It  is  shown  that  processing  with 
multiple  auxiliary  vectors  falls  under  well  known  blocking-matrix  processing  principles.  The 
proposed  blocking  matrix,  however,  is  data  dependent,  adaptively  generated,  and  no  data  eigen 
analysis  is  involved. 

While  the  issues  treated  refer  to  general  adaptive  detection  procedures,  the  presentation  is 
given  in  the  context  of  joint  space-time  adaptive  processing  for  array  radars. 
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JOINT  DOMAIN  SPACE-TIME  ADAPTIVE  PROCESSING 
WITH  SMALL  TRAINING  DATA  SETS 


Dimitris  A.  Pados 


I.  Introduction 

Detection  of  a  signal  in  Gaussian  noise  falls  under  the  binary  hypothesis  testing  version  of  the 
general  Gaussian  problem  and  lies  in  the  foundation  of  the  theory  of  optimum  detection.  Opti¬ 
mum  data  processing  in  the  Bayes,  Neyman- Pearson,  maximum  output  Signal-to-Noise- Ratio 
(SNR),  minimum  Mean-Square-Error  (MSE),  or  Minimum- Variance-Distortionless-Response 
(MVDR)  sense  are  all  well  known,  and  lead  to  scaled  versions  of  the  same  linear  filter  solution 
[1],  [2],  traditionally  called  Wiener  or  colored  noise  Matched  Filter  (MF).  Calculation  of  this 
filter  requires  knowledge  of  the  inverse  covariance  matrix  of  the  noise  process. 

In  the  context  of  an  array  radar  application  with  M  antenna  elements  (spatial  channels) 
and  N  pulses  per  coherent  processing  interval  (CPI),  optimum  signal  detection  in  the  presence 
of  spatial  and  temporal  Gaussian  noise  requires  joint  space-time  matched  filtering  in  the  M N 
complex  vector  space  [3].  Since  prior  knowledge  of  the  noise  covariance  matrix  is  not  available, 
the  maximum-likelihood  sample  average  estimate  is  often  used,  developed  from  K  noise  only 
vector  samples  that  correspond  to  distinct  range  cells.  This  is  the  so  called  secondary  data  set. 
Then,  the  inverse  of  the  sample-matrix  is  considered  as  an  estimate  of  the  inverse  covariance 
matrix.  This  approach  is  known  as  the  Sample-Matrix-Inversion  method  (SMI)  and  in  [4] 
it  was  shown  to  outperform  the  recursive  least-mean-squares  (LMS)  adaptive  implementation 
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of  the  matched  filter  processor  in  terms  of  convergence  rate  and  small-sample  output  SNR 
characteristics.  Still,  it  was  found  that  K  >  2MN  independent  and  identically  distributed 
(i.i.d.)  training  data  samples  are  needed  to  maintain  with  probability  1/2  a  loss  lower  than 
or  equal  to  3dB  compared  to  the  ideal  MF.  Simple  variance  normalization  of  the  MF  decision 
statistic  led  to  a  Constant  False  Alarm  Rate  (CFAR)  test  [5]  for  Neyman- Pearson  detection. 

System  optimization  in  the  generalized  likelihood  ratio  (GLR)  sense  was  pursued  in  [6].  The 
resulting  test  statistic  offers  embedded  CFAR  behavior,  converges  to  the  ideal  MF  solution  at 
least  in  probability  as  the  number  of  secondary  data  K  grows,  and  it  is  shown  in  [5]  to  outper¬ 
form  the  SMI  approach  for  K  =  2 MN,  except  for  high  SNR  regions.  Extended  performance 
analysis  and  studies  of  the  sidelobe  behavior  of  the  GLR  filter  are  carried  out  in  [7]. 

It  is  important  to  note  that  the  SMI  and  the  GLR  test  are  both  asymptotically  optimum 
to  the  extent  that  they  converge  in  a  probabilistic  sense  to  the  optimum  ideal  MF  as  the  size 
of  the  secondary  data  set  grows  to  infinity.  However,  for  finite  sample  support  no  optimality 
can  be  claimed  in  either  case  and  superior  probability  of  detection  performance  for  a  fixed  false 
alarm  rate  is  theoretically  possible  by  other  filtering  means.  Moreover,  both  methods  share 
the  need  to  invert  the  sample  covariance  matrix  of  the  noise  process  which  implies:  (i)  general 
computational  complexity  of  order  ( MN )3,  (ii)  secondary  data  set  of  at  least  K  =  2MN 
samples  to  maintain  with  probability  1/2  a  loss  compared  to  the  optimum  rule  of  no  more 
than  3dB,  and  (tit)  K  >  MN  to  ensure  nonsingularity  of  the  sample  covariance  matrix  with 
probability  1  [6].  Property  (ii)  is  due  to  characteristics  of  the  Wishart  distribution  that  the 
sample  covariance  matrix  is  assumed  to  follow  under  the  i.i.d.  and  Gaussian  assumptions. 
Property  (iii)  results  from  linear  algebraic  considerations  of  basis  vectors  needed  to  represent 
any  MiV-tuple  vector. 
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Arguably,  in  airborne  surveillance  systems  the  training  data  size  requirements  make  the 
practicality  of  these  approaches  questionable  even  for  moderate  values  of  M  and  N.  This  is 
particularly  true  if  we  consider  a  highly  non-stationary,  non-homogeneous  [8]  operating  envi¬ 
ronment,  typically  encountered  in  practice.  In  this  context,  it  is  tempting  to  consider  abolishing 
the  concept  of  joint  space-time  processing  and  pursue  separate  (“disjoint”  or  “cascade”)  SMI 
or  GLR  detection  in  the  M-dimensional  space  and  the  N-dimensional  time  domain.  It  is  easy, 
however,  to  identify  realistic  cases  where  the  performance  of  these  cascade  schemes  falls  unac¬ 
ceptably  below  that  of  the  joint  domain  approaches  [9]. 

In  this  present  work  the  principle  of  joint  space-time  (S-T)  processing  is  preserved  but 
instead  of  seeking  an  adaptive  implementation  of  the  optimum  linear  filter  in  the  CMN  complex 
space,  suboptimum  simpler  solutions  are  pursued.  The  objective  is  linear  space-time  adaptive 
processing  with  low  computational  optimization  complexity  and  sample  support  near  or  below 
the  space-time  product  MN.  In  this  context,  the  target  vector  of  interest  is  accompanied 
by  a  carefully  designed,  data  dependent,  orthogonal  auxiliary  vector  for  joint  S-T  processing 
with  a  single  complex  degree  of  freedom.  This  approach  is  inductively  extended  to  generate 
a  sequence  of  conditionally  optimized  auxiliary  vectors  -orthogonal  to  each  other  and  to  the 
target  of  interest-  for  adaptive  S-T  processing  with  any  desired  number  of  complex  degrees 
of  freedom  less  than  or  equal  to  MN  —  1.  Interestingly  enough,  we  show  that  processing 
with  multiple  auxiliary  vectors  falls  under  general  blocking-matrix  processing  principles  [10]- 
[14].  The  proposed  blocking  matrix,  however,  is  data  dependent  and  no  data  eigen  analysis 
is  involved.  Numerical  and  simulation  results  included  in  this  work  support  the  theoretical 
arguments  and  promote  the  new  method  as  a  desirable  low  computational  cost  alternative  for 
joint  space-time  adaptive  processing  when  very  small  secondary  data  training  sets  are  available. 
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The  SMI  and  GLR  tests  are  treated  as  performance  (probability  of  detection)  and  computational 
complexity  benchmarks. 

The  rest  of  this  manuscript  is  organized  as  follows.  Section  II  introduces  the  system  and  sig¬ 
nal  models.  The  principles  of  Auxiliary- Vector  filtering  and  the  new  test  statistics  are  developed 
and  studied  in  Section  III.  Probability  of  detection  comparisons  with  the  ideal  Matched-Filter, 
the  adaptive  Matched-Filter  and  the  Generalized- Likelihood-Ratio  test  are  carried  out  in  Sec¬ 
tion  IV  for  a  few  representative  cases. 


II.  Signal  Model  and  Background 

Consider  a  narrowband  uniform  linear  radar  array  with  M  antenna  elements  (subarrays  or 
spatial  channels).  Also,  assume  that  each  element  collects  the  complex  (I/Q)  return  of  a  series 
of  N  coherent  pulses  for  some  given  range  cell  k  -  1, . . . ,  Kmax  =  TPRI/T,  where  TPR[  is  the 
pulse  repetition  interval  and  T  is  the  pulse  duration.  We  organize  the  received  data  in  the 
form  of  a  matrix  XmxN,  where  X(m,  n),  m  =  1, . . . ,  M,  n  =  1, . . . ,  N,  denotes  the  m-element, 
n-pulse  signal  sample.  The  objective  is  to  cope  with  system  and  surrounding  disturbances 
and  detect  in  Xm*n  the  presence  of  a  desired  signal  of  unknown  amplitude.  Without  loss 
of  generality  and  for  notational  simplicity,  we  consider  a  “vectorized”  form  of  XmxR,  where 
Vec(XMXN)  =  X-mnx i  is  constructed  by  sequencing  all  matrix  columns  in  the  form  of  a  vector. 
From  now  on,  bold  variables  indicate  vectors  in  the  CMN  complex  space  unless  otherwise 
specified. 

We  begin  by  casting  the  detection  problem  in  the  context  of  binary  hypothesis  testing.  We 
denote  the  disturbance  only  hypothesis  by  H0  and  the  target  plus  disturbance  hypothesis  by 
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Ho  :  X  =  J  +  C  +  N 


Hi. 


Hj  :X  =  aV  + J  +  C  +  N.  (1) 

In  (1),  J  represents  a  mixture  of  L  broadband  directional  interferers  (jammers)  in  the  far  field, 
where  J  =  Vcc(Jmxn)  and 

J(m,n)  =  (2) 

/=i 

n  =  l,...,N,  m  =  1,...,M. 


We  assume  that  =  1  ,...,£,  is  complex  white  Gaussian  distributed  to  account  for 

channel  fading  at  the  pulse-repetition  frequency  as  in  the  model  of  [15]  and  [16].  The  antenna 
element  spacing  is  d  and  the  radar  carrier  wavelength  is  A.  The  direction  of  arrival  (DOA) 
9i,  l  =  1, . . . ,  L,  is  assumed  to  be  uniformly  distributed  in  [— tt/2,  7t/2].  We  find  it  convenient  to 
define  the  spatial  frequency 


a  sinOid 

Ji  =  — i — , 


/=!,.. .,L, 


(3) 


and  assume  that  /,  is  uniformly  distributed  in  [-0.5, 0.5]  with  proper  selection  of  d  and  A.  In 
addition,  C mnxi  in  (1)  accounts  for  colored  Gaussian  noise  with  covaxiance  matrix  Rc  and 
corresponds  to  a  radar  clutter  region.  Spatially  and  temporally  white  disturbances  are  denoted 
by  N.  The  signal  or  “steering  vector”  of  interest,  V,  is  present  in  X  under  hypothesis  Hi  only. 
If  denotes  the  Hermitian  operation  on  V ,  then  without  loss  of  generality  we  assume  that 
=  1  such  that  all  energy  signal  characteristics  are  absorbed  in  the  unknown  complex 
amplitude  constant  a.  For  completeness,  if  V  =  Vec(VMxN)  then 


V(m,n ) 


-  1  eMm-l)^+;2,(»-l)^ 

Vmn 
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In  (4),  6a  €  [— 7r/2,  tt/2]  is  the  angle  of  arrival  of  the  signal  (target)  of  interest,  v  is  the  target 
radial  velocity,  and  /pp  is  the  radar  pulse  repetition  frequency.  Once  again,  it  is  convenient  to 
define  the  spatial  target  frequency 


r  A  sin9sd 

js  =  - 


(5) 


and  the  “normalized  Doppler”  target  frequency 


a  2i/ 

A/pp 


In  this  case  we  assume  that  fa,fo  €  [—0.5, 0.5]. 


(6) 


Given  a  data  vector  X  that  corresponds  to  some  range  cell  k  €  {1, . . . ,  Kmax},  the  objective 
is  to  decide  in  favor  of  Ho  or  H\  in  a  way  that  maximizes  the  probability  of  detection  Pd  = 
Pr{Hi  deddtd\H\  true }  subject  to  a  given  false  alarm  constraint  PFA  =  Pr{Hi  decided\H0  true}  < 
p.  The  optimum  decision  rule  is  well  known  [1],  [3]  and  of  the  form 


Hi 


Ho 


(7) 


where  r  >  0  is  the  threshold  parameter  to  be  determined  according  to  the  condition  Pra  =  P 
and  w  is  the  linear  filter  defined  by 


w  =  fcR  V, 


(8) 


where  b  is  an  arbitrary  positive  scalar.  In  (8),  R  =  EHo{XXH}  where  EHl{-}  denotes  the  sta¬ 
tistical  expectation  operation  under  if,,  i  =  0, 1.  Then  the  variance  under  H0  of  the  test  statistic 
wHX  is  Vartfofvv^X}  =  VHR-1V  which  implies  that  the  modified  test  statistic  is 

VV^R"1  V 


vhr-»x  |2 


"R-'V 


is  chi- 


zero-mean  unit- variance  complex  Gaussian  distributed.  As  a  consequence 
square  distributed  with  two  degrees  of  freedom  and  leads  to  the  familiar  CFAR  (constant  false 


7-8 


alarm  rate)  optimum  decision  rule 


jV^R^XI2  > 
V*R"lV  <  A’ 


(A  >  0). 


(9) 


Substitution  in  (9)  of  the  sample-average  covariance  matrix  estimate  R  —  J2k=i  X^Xj^  from 
K  data  samples  from  H0  defines  the  so  called  CFAR  adaptive  matched  filter  (AMF)  detector 
[5].  In  contrast,  we  recall  that  the  generalized  likelihood  ratio  (GLR)  test  of  [6]  and  [7]  is 

|VHR_1X|2  > 

VHR-1V(1  +  iXWR-iX)  <  7’  (7  >  0)' 

In  the  following  section  we  develop  new  decision  statistics  that  maintain  the  principle  of 
linear  filtering  for  signal  detection  in  Gaussian  disturbance.  Compared  to  the  AMF  and  GLR 
approaches  we  seek  computational  savings  and  superior  performance  for  optimization  with  small 
secondary  data  sets  ( K  <  MN). 

III.  Adaptive  Auxiliary- Vector  Detection 

To  build  progressively  a  new  decision  statistic  we  begin  with  the  introduction  of  an  arbitrary 
fixed  “auxiliary”  vector  Gi  in  CMN  that  is  orthonormal  with  respect  to  the  normalized  target 
vector  of  interest  V : 

GiHV  =  0  and  G1HG1  =  1.  (11) 

Next,  we  consider  S-T  adaptive  processing  with  a  single  joint  S-T  degree  of  freedom  in  the  form 
of  w^yX,  where  [17] 

wAV  =  V  -  piGi ,  HieC.  (12) 
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With  an  arbitrary  but  fixed  auxiliary  vector  Gx  that  satisfies  the  constraints  in  (11),  the 
filter  in  (12)  can  be  optimized  with  respect  to  the  complex  scalar  /zx  only.  The  MS-optimum 
value  of  /zx  can  be  identified  from  two  different  -yet  equivalent-  points  of  view.  Since  w^y  is  by 
construction  distortionless  in  the  V  direction  (wAVV  =  1),  we  may  seek  the  value  of  /zx  that 
minimizes  the  output  variance  (energy)  £{|w^X|2}.  An  equivalent  interpetation  is  to  look  for 
the  value  of  /zx  that  minimizes  the  MSE  E{(VHX  -  p*1G1HXf}  between  the  S-T  main-beam 
processed  data  VHX  and  the  orthogonal  auxiliary- vector  processed  data  /z^GfX.  The  solution 
to  the  first  minimum  output  variance  problem  can  be  obtained  by  direct  differentiation  of  the 
variance  expression.  The  solution  to  the  second  MSE  problem  can  be  found  in  the  context  of 
linear  MS-optimum  estimation  [2].  The  solution  is,  of  course,  identical  in  both  cases  and  it  is 
identified  by  the  following  proposition. 


Proposition  1  The  value  of  the  complex  scalar  /zx  that  minimizes  the  output  variance  of  the 
filter  wAy  =  V  —  /zxGx  or  minimizes  the  mean  square  error  between  the  main-beam  processed 
data  VHX  and  the  auxiliary-vector  processed  data  /z*  G[^X  is 


hi  = 


GfRV 

G?RGX 


□ 


(13) 


Proposition  1  identifies  the  optimum  value  of  /zx  for  any  fixed  auxiliary  vector  Gx.  We  now 
turn  our  attention  to  the  selection  of  an  auxiliary  vector  subject  to  an  appropriately  chosen 
criterion.  We  understand  that  there  exists  a  vector  Gx  in  the  subspace  orthogonal  to  V  for 
which  wAV  and  the  optimum  filter  w  in  (8)  coincide.  This  solution,  however,  is  not  desirable 
since  it  reverts  to  processing  with  full  MN  degrees  of  freedom  and  requires  inversion  of  the 
pertinent  S-T  covariance  matrix.  The  selection  criterion  for  the  auxiliary  vector  Gx  that  we 
propose  in  this  work  is  motivated  by  the  MSE  interpretation  of  the  filter  in  (12)  and  it  is  the 
maximization  of  the  magnitude  of  the  crosscorrelation  between  the  V  processed  data  VHX  and 
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the  auxiliary- vector  processed  data  GfX,  subject  to  the  constraints  in  (11): 

Gx  =  argmax  |E{VffX(G(/X)*}|  =  arg  max  |  V^RGx  | ,  (14) 

subject  to  Gf  V  =  0  and  Gf  Gx  =  1 . 

We  note  that  both  the  criterion  function  |V^RGj|  and  the  constraints  are  phase  invariant.  In 
other  words,  if  G'  maximizes  IV^RG^  under  the  given  constraints,  so  does  G'e“J^  for  any 
phase  <j>.  Therefore,  without  loss  of  generality,  we  may  identify  the  unique  auxiliary  vector  that 
is  a  solution  to  the  constraint  optimization  problem  in  (20)  and  places  VHRG!  on  the  non¬ 
negative  real  line  (VhRGi  >  0).  The  following  proposition  identifies  the  optimum  auxiliary 
vector  according  to  our  criterion.  The  proof  is  omitted  due  to  lack  of  space. 

Proposition  2  The  auxiliary  vector 

r  _  RV  —  (VHRV)V 

1  ~  ||RV  -  (Vi7RV)V||  (15) 

maximizes  the  magnitude  of  the  crosscorrelation  between  the  main-beam  processed  data  VHX 
(w.l.o.g.  VHV  =  1)  and  the  auxiliary  vector  processed  data  GfX,  |VhRGi|,  subject  to  the 
constraints  G^V  =  0  and  G^Gi  =  1.  □ 

Propositions  1  and  2  identify  completely  the  S-T  adaptive  filter  in  (12).  To  generalize  these 
results,  we  may  consider  a  filter  wc  consisting  of  P,  1  <  P  <  MN  -  1,  weighted  auxiliary 
vectors  that  are  orthonormal  with  respect  to  each  other  and  to  the  normalized  target  vector  V: 

A  P 

(16) 

t=l 

The  sequence  of  auxiliary  vectors  Gt-  and  weights  /x,-,  i  =  1, . . . ,  P,  are  conditionally  optimized 
and  defined  inductively.  First,  Gi  and  /ij  are  optimized  exactly  as  in  (15)  and  (13).  Next,  we 
upgrade  the  “main-beam”  to  V  —  /xjGj  and  we  seek  an  auxiliary  vector  G2  that  maximizes 
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the  crosscorreiation  magnitude  |(V  -  subject  to  the  orthonormality  constraints 

=  0,  G^Gj  =  0,  and  G^G2  =  1.  Arguing  as  in  Proposition  2  we  find 

q  =  R(V-/* iQi)  -  [V^(V  - /xiGi)]V  -  [GfR(V  —  ^iG^jGt 

||R(V  -  /xjGx)  -  [VHR(V  -  /xjGxllV  -  [G"R(V  -  ^G^GxIl  '  (17) 

The  conditionally  MS-optimum  value  of  p2  given  Gi,  /ij,  and  G2  is 


^2 


GfRCV-^GQ 

Gf  RG2 


(18) 


To  complete  the  recursive  construction  of  the  filter  wc  in  (16)  let  us  assume  that  Gt,  m  are 
defined  for  i  l,...,p,  1  <  p  <  P  1.  Then,  with  umain-beam”  V  —  ptG,',  the  auxiliary 
vector  Gp+1  that  maximizes  the  crosscorrelation  magnitude  |(V  -  Y$=i  p.G.^RGp+x  |  subject 
to  the  orthonormality  constraints  is 


q  =  R(v  -  EL,  pA)  -  [v"R(v  -  el,  /‘.g,))v  -  EL.iQf R(v  -  EL,  /*.g,)]gj 
,+1  IIR(V  -  EL,  P.G.)  -  [VffR(V  —  EL,  P,G;)]V  -  EL,[Gf  R(V  -  EL,  /,,G,)]G)-|| 

(19) 

The  conditionally  MS-optimum  coefficient  fip+1  becomes 


t*P+ 1 


G"+,R(V  -  EL,  bS,) 

G«,RGP+1 


(20) 


Equations  (19)  and  (20)  create  a  sequence  of  conditionally  optimized  auxiliary  vectors  and 


conditionally  optimized  weights  and  identify  completely  the  filter  under  consideration  wc  in 
(16).  To  seek  further  improvements  in  the  form  of  unconditional  weight  optimization  and  to 
relate  with  existing  literature,  we  find  it  convenient  to  organize  the  auxiliary  vectors  in  the 
form  of  an  MN  x  P  matrix  B  where  Gp,  p  =  1, . . . ,  P,  constitutes  the  p-th  column  of  B: 


Bmwxp  —  [Gi  G2  . . .  Gp], 


With  this  setup  the  orthonormality  conditions  translate  to 


BHV  =  0pxl  and  BWB  =  lpxp, 


(22) 
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and  the  overall  linear  filter  wc  in  (16)  can  be  written  compactly  as 

w c  =  v  -  =  V  -  B fi.  (23) 

1=1 

We  call  B  a  1 blocking ”  matrix  because  it  blocks  signal  components  in  the  S-T  direction  of 
interest  V.  Blocking  matrix  processing  techniques  have  been  of  considerable  interest  in  array 
processing  applications  in  the  past  [10].  In  fact,  some  recent  research  efforts  focus  on  the 
selection  of  the  blocking  processor  based  on  eigenvector  metrics  [11]- [12]  or  data  independent 
Discrete- Fourier  (DFX)  or  Walsh-Hadamard  transforms  (WHT)  [18].  In  this  present  work, 
the  formulation  of  the  blocking  matrix  B  in  (21)  is  data  dependent  with  low  computational 
complexity  and  no  eigen  analysis  involved.  With  respect  to  the  conditionally  optimized  weights 
in  the  form  of  the  vector  n,  an  alternative  filter  design  that  involves  the  unconditional  MS- 
optimum  vector  weight  value  n°  can  be  sought: 

wB  =  V  -  B fi° .  (24) 

There  are  two  equivalent  interpretations  for  the  MS-optimum  value  of  the  weight  vector  fi°. 
Since  the  overall  blocking  matrix  filter  wB  in  (24)  is  by  construction  distortionless  in  the  V 
direction  of  interest  (wB*V  =  1),  we  may  search  for  the  weight  vector  n°  that  minimizes 
the  output  variance  (energy)  £{|wBHX|2}.  Equivalently,  we  may  instead  identify  the  vector 
fi  that  minimizes  the  MSE  E{(VHX  —  fi°^ B^X)2}  between  the  main-beam  processed  data 
VHX  and  the  blocking-matrix  processed  data  fioHBHX.  The  solution  to  the  first  problem 
statement  can  be  obtained  by  direct  differentiation  of  the  variance  expression.  The  theory  of 
linear  MS-optimum  estimation  [2]  offers  the  solution  to  the  second  problem.  The  solutions,  of 
course,  coincide  and  the  optimum  weight  vector  is  of  the  form  of  a  generalized  sidelobe  canceler 
identified  by  Proposition  3  below.  This  result  is  a  generalization  of  Proposition  1. 
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Proposition  3  The  P-dimension,  1  <  P  <  MN  —  1,  complex  weight  vector  n°  that  minimizes 
the  output  variance  of  the  filter  wB  =  V  -  B \x°  or  minimizes  the  mean  square  error  between 
the  main-beam  processed  data  VHX  and  the  blocking-matrix  processed  data  fioHBHX  is 

»°  =  [BwRB]-1BffRV .  □  (25) 

We  conclude  this  section  with  a  few  observations.  Blocking  matrix  processing  with  P  joint 
S-T  degrees  of  freedom  requires  inversion  of  the  P  x  P  blocked-data  covariance  matrix  BHRB, 
as  seen  by  (25).  The  output  variance  of  the  ideal  filter  wB  as  a  function  of  the  blocking  matrix 
B  can  be  seen  to  be 

VARb  =  £{|wbhX|2}  =  VhRV  -  VhRB[B//RB]-1BhRV.  (26) 

By  construction,  comparison  with  the  output  variance  of  the  ideal  MF  filter  in  (8),  VARMF  = 
VHR_1V,  and  the  output  variance  of  the  ideal  wc  filter  in  (16),  VARC,  shows  that  VARMF  < 
VARb  <  V ARC  and  V ARmF  =  V ARb  if  P  —  MN  —  1,  while  VARb  =  V ARC  \i  P  =  \. 

For  the  proposed  filters  wc  in  (16)  and  wB  in  (24)  and  any  number  of  auxiliary  vectors 
^  P  —  MN  —  1,  the  decision  statistics  and  are  0— mean,  unit- variance 

complex  Gaussian.  Then, 


I  w"X  j2  > 
wfRwc  < 


A  and 


I  wB"X  |2  > 
wbwRwb  < 

H0 


(27) 


define  CFAR  tests  in  parallel  to  the  optimum  matched-filter  test  in  (9).  In  adaptive  implemen¬ 
tations  of  wc  and  wB,  say  wc  and  wB,  the  covariance  matrix  R  needed  in  (19),  (20),  and  (25) 
is  estimated  through  the  maximum  likelihood  sample-average  procedure  as  usual: 

X(*)X(*)».  (28) 

k= 1 
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The  tests  are  asymptotically  CFAR  (as  K  *  oo,  Ai,  A2  — *  A  in  (9)  for  the  optimum  normalized 
matched-filter  test  for  any  given  false  alarm  rate  PFA).  For  finite  training  sets  they  qualify  as 
Cell-Averaging  CFAR  (CA-CFAR),  as  seen  by  the  test  denominators. 

IV.  Numerical  and  Simulation  Comparisons 

In  this  section  we  support  and  illustrate  the  preceding  theoretical  developments  through  four 
representative  case-studies  based  on  the  hypothesis  testing  problem  in  (1).  In  all  cases  we 
assume  presence  of  a  mixture  of  3  (three)  broadband  interferes  with  Jammer-to-Noise-Ratio 
JNR  =  I  «  35 dB,  l  =  1,2,3.  The  jammers  follow  the  model  in  (2)  and  the  spatial 
frequency  /  =  1,2,3,  is  randomly  drawn  from  the  uniform  [-0.5, 0.5]  distribution.  The 
“peak”  clutter  (colored  Gaussian  noise)  to  noise  ratio  is  fixed  at  CNR  =  4  =  40 dB.  The 
false  alarm  rate  is  set  at  PFA  =  .01  and  for  simulation  purposes  threshold  and  probability  of 
detection  estimates  Pd  are  based  on  10, 000  samples  from  Ho  and  H\  respectively.  All  presented 
results  are  averages  over  100  independent  Monte-Carlo  runs  for  arbitrarily  chosen  target  vectors 
and  jammers.  PD  values  are  plotted  as  a  function  of  the  total  SINR  defined  by 

SI  NR  =  |  a  |2  VhR-1V.  (30) 

We  assume  a  radar  array  system  with  M  =  5  antenna  elements  (channels)  and  N  -  12  pulses 
per  coherent  processing  interval  (CPI).  All  studies  include  the  ideal  matched  filter  as  a  reference 


point,  and  the  adaptive  matched  filter,  the  GLR  test,  the  adaptive  wc  filter  in  (16)  with  one 
and  ten  auxiliary  vectors  (P  =  1  and  P  =  10),  and  the  adaptive  wb  filter  in  (24)  with  thirty 
auxiliary  vectors  (P  =  30),  all  for  secondary  data  sets  of  size  K  =  MN  =  60. 

Figures  1  through  4  present  probability  of  detection  versus  SINR  results  for  four  different 
clutter  scenarios  generated  as  in  [19]  with  normalized  one-lag  temporal  clutter  correlation  0.4, 
0.6,  0.97,  and  0.999,  respectively.  The  eigen  profiles  of  the  corresponding  true  clutter  covariance 
matrices  are  shown  in  Figs.  5  through  8.  We  again  note  that  the  algorithmic  developments 
in  this  work  do  not  involve  data  eigen  analysis.  However,  the  eigen  profiles  provide  an  under¬ 
standing  of  the  need  for  an  increased  number  of  auxiliary  vectors  with  unconditional  weight 
optimization  when  operating  in  the  presence  of  high  temporal  correlation  clutter. 

Relatively  low  temporal  clutter  correlation  (low  pulse-repetition  frequency  fPR  and/or  high 
clutter  velocity)  results  to  benign  eigenvalue  distributions  as  seen  in  Figs.  5  and  6.  In  such  cases 
the  proposed  filter  wc  in  (16)  offers  significant  performance  gains  even  with  very  few  auxiliary 
vectors  and,  therefore,  minimal  computational  cost. 

For  higher  temporal  clutter  correlation  (high  pulse-repetition  frequency  fpR  and  low  clutter 
velocity)  the  proposed  filter  wB  in  (24)  becomes  increasingly  more  effective.  For  the  somewhat 
typical  case  of  one-lag  temporal  clutter  correlation  0.999  in  Fig.  4,  performance  gains  in  excess 
of  7 dB  compared  to  the  GLR  and  adaptive  MF  counterparts  can  be  seen.  This  is  particularly 
true  for  high  SINR  values  and  detection  with  probability  close  to  1. 
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Figure  1:  Pd  vs  SINR  (one-lag  temporal  clutter  correlation  0.4). 
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Figure  2:  Pd  vs  SINR  (one-lag  temporal  clutter  correlation  0.6). 
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Figure  3:  Pd  vs  SINR  (one-lag  temporal  clutter  correlation  0.97). 
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Figure  4:  Pd  vs  SINR  (one-lag  temporal  clutter  correlation  0.999). 
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Figure  5:  Eigen  profile  of  clutter  covariance  matrix  in  Fig.  1. 
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Figure  6:  Eigen  profile  of  clutter  covariance  matrix  in  Fig.  2. 
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Figure  7:  Eigen  profile  of  clutter  covariance  matrix  in  Fig.  3. 


Figure  8:  Eigen  profile  of  clutter  covariance  matrix  in  Fig.  4. 
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Abstract 

System  invasion  is  a  common  phenomenon  in  today's  world  specially  after  the  concept  of  information 
sharing  paved  its  way  into  the  technical  and  business  world.  This  certainly  affects  databases  and  brings  them  to  an 
unstable  and  inconsistent  state.  It  is  required  to  recover  and  make  the  database  operable  as  soon  as  such  an  attack 
is  detected.  However,  extensive  use  of  log  is  required  during  the  damage  assessment  and  recovery  process.  Log 
access  time  becomes  a  significant  cost  in  this  process  due  to  the  massive  amount  of  data  stored  in  the  log.  and  the 
page  I/O  costs  involved  with  it.  In  this  research,  we  have  developed  algorithms  to  cluster  the  log  based  on 
transaction  read-from  relationships.  This  helps  in  eliminating  the  requirement  of  accessing  the  entire  log,  thus 
considerably  reducing  the  damage  assessment  and  recovery  time.  We  prove  that  only  one  of  these  clusters  will  be 
accessed  during  damage  assessment  and  recovery  process.  Through  simulation  we  have  confirmed  that  our  model 
outperforms  the  traditional  logging  method  in  assessing  the  damage. 
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Information  Warfare:  Faster  Damage  Assessment  Through  Efficient  Logging 


Brajendra  Panda 


1.  Introduction 

Sharing  brings  growth.  This  truth  brought  into  picture  the  concept  of  resources  and  information  sharing.  To 
sustain  in  the  incredibly  changing  technological  world,  we  need  to  share  information  with  others.  Computers  are 
probably  the  most  powerful  information  sharing  devices  in  today's  world  and  thanks  to  the  Internet  to  make  the 
process  of  information  sharing  even  faster  and  easy.  But  every  virtue  brings  a  vice  along  with  it.  When  we  use  our 
system  to  share  information  with  others  via  network,  we  leave  the  door  open  for  hackers  to  invade  our  system.  It  is 
extremely  difficult  to  find  and  close  all  security  flaws  in  the  system.  Hackers  are  always  in  search  of  new  ways  to 
prevail  over  the  system  security.  Password  sniffing  and  session  hijacking  are  among  various  other  means  of 
intruding  into  a  system.  In  such  cases,  the  system  under  attack  will  not  be  able  to  detect  an  attacker  from  a 
legitimate  user. 

Defensive  information  warfare,  which  prepares  the  system  to  withstand  such  attacks  and  provide  system 
integrity,  is  based  on  the  model  of  prevention,  detection,  and  recovery.  A  lot  of  research  has  been  performed 
including  strict  access  control  mechanisms  to  protect  the  system  from  unauthorized  users.  Since  prevention  issues 
are  outside  the  scope  of  this  research,  we  would  not  reference  them  here.  There  are  various  intrusion  detection 
techniques  available  to  observe  suspicious  activities  of  users,  and  also  detect  any  unauthorized  access.  Among 
them,  a  knowledge-based  approach  is  presented  in  [7],  while  a  statistical  approach  is  offered  in  [8].  McDermott  and 
Goldschalg  have  discussed  a  method  called  storage  jamming  ([10],  [11]),  that  uses  fake  data  termed  as  “honey  pots” 
to  attract  attackers  and  flush  them  out.  Several  other  techniques  have  been  discussed  in  [9].  However,  not  all  these 
methods  detect  an  attack  as  it  occurs.  In  most  cases,  the  attacker  enters  the  system  and  accesses  data  as  a  valid  user. 
Thus,  any  update  performed  by  the  attacking  transaction  is  made  permanent  when  the  transaction  commits. 
Therefore,  the  damage  can  spread  to  other  parts  of  the  database  through  legitimate  users  as  they  update  fresh  data 
after  reading  any  damaged  data  ([1],  [5]).  Consequently,  timely  recovery  of  the  database  is  crucial  to  stop  the 
cascading  effect  of  the  damage.  Since  operations  of  each  transaction  is  stored  in  system  log,  and  recovery 
algorithms  that  are  designed  for  defensive  information  warfare  scenarios  require  that  the  log  must  never  be  purged, 
the  size  of  the  log  becomes  massive.  Therefore,  the  log  access  time  during  damage  assessment  and  recovery 
becomes  a  significant  factor. 

In  this  research,  we  present  algorithms  to  divide  the  log  into  several  clusters  based  on  transaction 
dependencies.  We  prove  that  only  one  of  these  clusters  will  be  accessed  during  damage  assessment  and  recovery 
process.  We  have  used  a  simulation  model  to  test  the  performance  of  our  method.  The  result  shows  dramatic 
improvement  over  the  time  required  to  access  traditional  logs. 
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The  rest  of  the  report  is  organized  as  follows.  In  section  2,  we  discuss  the  motivation  for  this  research. 
The  clustering  algorithm  and  its  proof  of  correctness  are  presented  in  section  3.  Section  4  offers  an  algorithm  for 
the  post  clustering  logging  of  transactions.  The  simulation  model  and  performance  comparisons  with  traditional 
logging  method  are  offered  in  section  5.  Section  6  concludes  the  report. 

2.  Motivation 

Traditional  recovery  mechanisms  as  discussed  in  [2],  [3],  [4],  and  [6]  have  their  scope  confined  to  system 
or  hardware  failures.  They  are  designed  to  flush  out  all  effects  of  all  non-committed  transactions  while  ensuring  that 
effects  of  all  committed  transactions  are  saved  in  the  stable  storage.  For  this  kind  of  recovery,  the  vitality  of  read 
operations  does  not  exist.  Recovery  is  possible  only  from  the  before  and  after  images  of  data  items.  However,  as 
stated  in  [12],  after  the  detection  of  an  attack,  effects  of  all  transactions  reading  directly  or  indirectly  from  the 
malicious  transaction  have  to  be  undone  along  with  that  of  the  malicious  transaction.  Then  all  affected  transactions 
must  be  re-executed  to  reflect  the  consistent  state  of  the  database.  Hence,  in  order  to  establish  read-from 
relationships  all  read  operations  of  transactions  need  to  be  stored  in  the  log  and  the  log  can  not  be  purged.  A 
recovery  protocol  based  on  data  dependency  approach  has  been  presented  in  [13]  and  [14],  which  allows  affected 
transactions  to  be  partially  re-executed  during  recovery,  therefore,  incurring  a  faster  recovery  process.  In  any  case, 
the  system  log  must  be  accessed  and  the  log  must  be  read  sequentially  starting  from  the  appearance  of  the  attacking 
transaction  in  the  log.  This  process  requires  unnecessary  read  of  all  unaffected  transactions  that  appear  after  the 
malicious  transaction,  thus  incurring  too  many  superfluous  page  I/Os.  Since  page  I/Os  play  the  most  dominating 
factor  in  data  processing,  avoidance  of  these  activities  as  much  as  possible  is  essential. 

3.  Log  Clustering 

In  this  section,  we  describe  a  method  for  dividing  the  log  into  several  clusters.  This  will  help  us  identify 
the  cluster  to  scan  during  damage  assessment  and  recovery  as  opposed  to  the  entire  log.  The  principal  idea  of  our 
model  is  that  the  part  of  the  log  that  contains  the  attacking  transaction  need  to  be  pinpointed,  and  only  the  clusters 
that  contain  affected  transactions  need  to  be  accessed.  This  will  save  considerable  amount  of  time  during  both 
damage  assessment  and  recovery.  Our  model  is  based  on  the  assumption  that  the  history  produced  by  transaction 
executions  is  strict.  Next,  we  define  some  terms,  which  are  used  in  our  model. 

Definition  1:  A  transaction  Tj  is  dependent  upon  transaction  7/  in  a  history  if  there  exists  a  data  item  x  such  that: 
(1)  7y  reads  x  after  Tj  updates  x,  and  (2)  every  transaction  that  updates  x  in  between  7/  updates  x  and  Tj  reads  x 
should  be  aborted  before  Tj  reads  it.  It  is  needless  to  mention  that,  since  the  history  is  strict,  7/  commits  before  Tj 
reads  x. 
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Definition  2:  If  a  transaction  Tj  is  dependent  upon  transaction  Tj,  then  we  denote  transaction  Tj  as  the 

dependent  transaction  and  Tj  as  the  precursor  transaction. 

Transaction  dependencies  can  be  established  by  maintaining  auxiliary  data  structures.  In  this  research,  we 
do  not  discuss  any  specific  ways  of  doing  so;  rather,  we  focus  on  the  recovery  model.  Our  first  algorithm  takes  the 
existing  log  as  input  and  uses  transaction  dependencies  to  divide  the  log  into  various  clusters.  However,  it  is 
possible  to  establish  dependency  relationships  while  the  transactions  are  executing.  With  this  second  approach,  it 
would  not  be  known  whether  a  particular  transaction  will  commit  or  abort.  If  a  transaction  aborts  and  it  is  already 
inserted  to  a  cluster,  then  the  transaction  must  be  removed  from  the  cluster,  thus  wasting  a  lot  of  effort.  An 
important  point  to  mention  here  is  that  if  Tj  is  dependent  on  transaction  Tj ,  then  transaction  Tj  can't  be  dependent 
on  transaction  Tj  since  it  would  then  violate  the  serializability  of  the  history. 

For  the  clustering  algorithm  to  keep  the  dependent  transactions  in  one  cluster,  we  use  a  temp_set  to  store 
the  transactions  until  a  dependency  is  found.  We  also  use  another  data  structure,  the  clusterjist,  where  each  cluster 
id  and  all  transaction  ids  belonging  to  the  corresponding  cluster  are  kept.  The  copy  operation  used  in  the  algorithm 
must  preserve  the  order  of  operations  as  they  appear  in  the  log.  This  will  preserve  the  original  dependency  by 
maintaining  the  partial  order  among  transaction  operations. 

1 .  Set  the  temp_set  =  { } ;  //Empty  set 

2.  Set  Cluster_list  =  {} ;  Set  Last_write  =  {} ; 

3.  Read  all  committed  transactions  from  the  log 

3.1  Check  each  operation  oj  of  every  transaction  7/  for  dependency 

3.1.1  If  no  dependency  found 

3 . 1 . 1 . 1  If  Tj  is  not  already  in  a  cluster 

Write  operation  oj  to  the  temp  set; 

3. 1.1.2  Else 

Write  operation  Oj  to  the  cluster  Tj  is  in; 

3.1.2  Else 

3. 1.2.1  If  both  T{  and  its  precursor  transaction,  say  Tj ,  are  in  temp_set 

Create  a  cluster; 

Copy  all  operations  of  Tj  and  7/  from  the  temp_set  to  the  cluster; 

Copy  operation  oj  to  the  cluster; 

3. 1.2.2  Else  if  only  the  precursor  transaction  Tj  is  in  a  cluster 

Copy  all  operations  of  Tj  from  tempset  to  the  same  cluster; 

Copy  operation  oj  to  the  cluster; 

3 . 1 .2.3  Else  if  7/  is  in  a  cluster  and  the  precursor  transaction  Tj  is  in  the  temp_set 
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Copy  all  operations  of  the  Tj  from  the  temp_set  to  the  same  cluster  T\  is  in; 

Copy  oi  to  the  cluster; 

3. 1.2.4  Else 

3.1. 2.4.1  If  both  Ti  and  Tj  are  in  the  same  cluster 

Copy  o(  to  the  cluster; 

3. 1.2.4. 2  Else 

Copy  all  operations  of  all  transactions  from  the  cluster  of  Tj  to  the  cluster  of  Tf, 
Copy  oi  to  its  cluster; 

Remove  the  cluster  Tj  was  in,  and  also  remove  the  cluster  entry  from  the 
cluster_list; 

3.1.3  If  oi  is  a  write  operation  (wz[jc])  for  data  item  x 
3 . 1 .3 . 1  If  jc  is  already  in  the  last_write  table 

Overwrite  the  entry  with  the  new  value  x,  7/  and  the  cluster  number  of  7/; 

3. 1.3.2  Else 

Insert  w/  [x]  and  the  cluster  number  of  7/  as  a  new  entry  in  the 
last_write  table 

Comments:  In  step  3. 1.1.1,  we  check  if  the  transaction  already  belongs  to  a  cluster.  This  is  possible  when  a  prior 
operation  of  7/  would  have  established  a  dependency  thus  inserting  7/  into  a  cluster.  Both  7/  and  Tj  may  be  in 
temp_set,  as  checked  in  step  3. 1.2.1,  when  Tj  is  independent  of  any  transaction,  and  none  of  the  prior  operations  of 
Ti  have  established  a  dependency  for  7*.  Step  3. 1.2.3  checks  if  the  precursor  transaction  Tj  is  in  the  temp  set,  but  7/ 
is  in  a  cluster.  This  could  occur  when  Tj  is  independent  of  any  transaction,  where  as  for  7),  another  pre-cursor 
transaction  was  found  earlier.  Due  to  the  same  reason,  i.e.,  when  7/  depends  on  multiple  transactions,  both  Ti  and  Tj 
may  be  found  in  two  different  clusters.  This  is  checked  in  step  3. 1.2.4. 2.  Also  note  that,  in  step  3. 1.2. 4. 2,  when  all 
operations  of  all  transactions  in  one  cluster  are  copied  into  another  cluster,  the  partial  dependencies  among  the 
transactions  in  both  clusters  are  preserved. 

Proof  of  Correctness 

Lemma  7:  All  operations  of  all  committed  transactions  are  checked  to  determine  the  dependency. 

Proof:  It  is  obvious  from  the  step  3.1  of  the  clustering  algorithm,  where  the  algorithm  checks  all  operations  in  the 
log.  Step  3  makes  sure  that  only  the  committed  transactions  from  the  log  are  taken  into  consideration  for  clustering 
algorithm. 

Lemma  2:  Any  transaction  that  has  a  dependency  relationship  with  another  transaction  ends  up  in  a  cluster. 

Proof  As  the  log  is  assumed  to  keep  all  the  operations  of  a  transaction,  we  check  operation  by  operation  of  all  the 
committed  transaction  to  establish  the  dependency.  If  Oi  is  an  operation,  then  in  3. 1.1.1  the  algorithm  checks 
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whether  that  operation  establishes  any  dependency.  If  it  is  not  and  it  is  not  in  a  cluster  already  it  stays  in  the 
temp  set  according  to  3. 1.1.1  until  Tj  establishes  any  kind  of  dependency  or  some  other  transaction  establishes 
dependency  with  Tj  making  it  a  precursor  transaction.  If  the  transaction  Tj  is  already  in  a  cluster  then  the  operation 
Oi  is  added  to  the  cluster.  3. 1.2.1  deals  with  the  case  if  dependency  takes  place  between  Tj  and  Tj  and  they  are  in 
temp_set,  a  new  cluster  is  built  and  these  two  transactions  are  kept  there.  In  subsequent  section  3. 1.2.2  and  3. 1.2. 3 
the  algorithm  deals  with  cases  where  only  precursor  transaction  is  in  a  cluster  and  only  dependent  transaction  is  in  a 
cluster  respectively.  In  3.1. 2.4.2  the  algorithm  deals  with  a  special  case  where  a  transaction  7y,  which  is  already  in  a 
cluster  establishes  dependency  with  another  transaction  Tfe  in  another  cluster.  In  that  case  it  is  suggested  to  merge 
the  two  clusters  into  one.  Therefore,  any  precursor  or  dependent  transaction  will  end  up  in  a  cluster. 

Lemma  3:  If  a  transaction  is  neither  dependent  nor  precursor  of  any  transaction  will  stay  in  the  temp_set. 

Proof:  Step  3. 1.1.1  checks  and  stores  each  operation  of  each  transaction  in  the  temp_set  if  it  is  not  establishing  any 
dependency  and  the  transactions  have  not  become  a  precursor  or  dependent  before.  The  entire  block  3.1.2  makes 
sure  that  any  transaction,  which  is  a  part  of  any  dependencies,  will  end  up  in  a  cluster.  The  above  argument 
suggests  that  at  the  end  of  the  execution  of  the  clustering  algorithm,  any  transaction  that  is  neither  a  precursor  nor 
dependent  transaction  will  reside  in  the  temp  _set. 

Lemma  4:  A  transaction  can  belong  to  only  one  cluster. 

Proof:  Imagine  a  transaction  Tj  is  already  in  a  cluster  with  its  precursor  transaction  Tj.  Then  an  operation  Oj  of 
transaction  Tj  establishes  a  dependency  with  transaction  Tfo  which  is  in  another  cluster.  Section  3. 1.2.4. 2  of  the 
clustering  algorithm  suggests  copying  all  the  operation  of  all  the  transactions  from  the  cluster  of  7>  to  the  cluster  of 
Tj  and  Tj.  Then  we  remove  the  cluster  of  Tfr  and  also  remove  the  cluster  from  the  cluster  list.  So  the  idea  is  to  merge 
the  clusters  if  a  transaction  in  one  cluster  belongs  to  multiple  clusters.  This  proves  our  claim  that  a  transaction  will 
belong  to  only  one  cluster. 

For  recovery  we  are  only  concerned  about  the  read-from  relationships  [3]  of  transactions,  so  it  is  important 
that  our  cluster  maintains  all  the  read-from  relationships  as  in  the  original  log.  The  following  theorem  establishes 
this. 

Theorem  1 :  The  read-from  relationships  among  transactions  remain  the  same  as  in  the  original  log. 

Proof:  Assume  that  transaction  Tj  has  a  read  from  relationship  with  transaction  T*  (  Tf  ->  Tj)  in  the  original  log,  Tj 
has  written  a  data  item,  say  x ,  which  is  read  by  Tj  without  any  committed  transaction  updating  x  in  between.  If  the 
cluster  that  contain  T,  and  Tj  shows  Tk  ->  Tj,  then  Tk  has  updated  x  after  T]  updated  x  and  before  Tj  read  x.  This 
means  the  copy  of  w£[x]  from  log  to  cluster  is  performed  after  the  copy  of  w([x]  and  before  that  of  rj[x].  Since  the 
operations  are  copied  in  the  same  order  as  they  appear  in  the  log,  the  order  of  their  appearance  in  the  log  is  wj[x\  < 
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wk\x]  <  rj[x].  This  contradicts  the  fact  that  Tf  has  read  from  relationship  with  transaction  Tj.  This  proves  the 
theorem. 

Theorem  2\  During  damage  assessment  and  recovery  only  one  cluster  will  be  accessed  for  one  attacker. 

Proof:  According  to  Lemma  4  a  transaction  will  remain  in  one  cluster  only.  If  a  transaction  establishes  a 
dependency,  we  merge  the  clusters  of  both  precursor  and  dependent  transactions.  According  to  Theorem  1  all  the 
read-ffom  relationships  are  kept  intact,  which  means  all  transactions  having  read-from  relationships  with  an 
attacking  transaction  will  be  in  the  same  cluster  as  the  attacking  transaction.  Therefore,  during  damage  assessment 
only  one  cluster  will  be  accessed  for  one  attacker. 

4.  Post-clustering  scenario 

After  the  execution  of  the  clustering  algorithm,  all  the  dependent  and  precursor  transactions  will  be  in  some 
clusters.  Clustersjist  will  have  the  updated  cluster  list  for  all  the  transactions.  Last_ write  table  will  contain  the  last 
updated  entries  for  every  data  item.  Temp  set  will  have  all  the  transactions,  which  are  neither  precursor  nor 
dependent.  Taking  all  this  change  into  consideration,  we  have  to  develop  a  model  for  the  post-clustering  scenario. 
Requirement  of  a  system  log  is  the  major  issue  during  the  post-clustering  operations  of  transaction  in  the  database  as 
there  will  be  no  system  log  in  the  post-clustering  scenario.  Functionality  of  the  system  log  is  of  primary  importance 
for  all  traditional  operations  in  a  database  and  for  all  the  traditional  recovery  techniques.  In  our  model,  we  have 
suggested  to  using  the  temp_set  as  the  active  log  and  temp_set  along  with  all  the  clusters  will  function  akin  to  the 
system  log  for  everyday  affair  of  the  database.  Both  active  and  committed  transactions  are  going  to  reside  in  the 
tempset  as  long  as  they  have  not  established  any  dependency.  Unlike  during  clustering,  we  are  going  to  deal  with 
active  transactions  here  in  the  post-clustering  scenario.  So  the  clustering  algorithm  is  not  quite  compatible  for  this 
changed  environment.  We  developed  another  algorithm  to  deal  with  this  situation,  which  is  presented  next. 

Post  Clustering  Algorithm 

This  new  algorithm  uses  the  old  data  structures  temp  set  and  last  write  table  used  in  the  clustering 
algorithm.  Along  with  this,  we  have  introduced  a  new  data  structure  called  precursor_list.  For  each  active 
transaction  we  suggest  to  maintain  a  precursor  list  in  the  transaction  buffer  whenever  any  dependency  is  established. 
Precursor  list  is  kept  in  the  buffer  until  the  transaction  aborts  or  commits  and  the  transactions  are  stored  in  a  cluster 
if  the  transaction  has  a  precursor,  otherwise  the  transaction  remains  in  the  temp  set.  If  transaction  commits  then 
only  its  operations  are  available  to  other  transactions.  As  the  history  is  strict,  no  transaction  can  read  or  write  a  data 
item  accessed  by  an  active  transaction.  The  following  algorithm  is  used  to  place  transactions  in  appropriate  clusters 
during  commit  time  ([2],  [3])  of  the  transactions. 

1.  For  each  operation  0[ 

1.1  If  oj  is  a  write  operation  W;[x] 
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Add  the  operation  to  the  temp_set; 

1 .2  If  of  is  a  read  operation  rj[x] 

Add  the  transaction  id  of  the  precursor  of  Tj  to  the  precursor  list  of  Tj  - 
Leave  it  in  temp_set; 

1 .3  If  of  is  an  abort  operation 

Remove  all  the  operation  of  transaction  7/  from  the  temp  set; 

Delete  the  precursor  list  of  Tx; 

1 .4  If  of  is  a  commit  operation 

Update  the  last_write  table  for  all  the  write  operations  done  by  7/  on  any  data  item. 

1.4.1  If  the  precursor  list  of  T\  is  not  empty 

1 .4. 1 . 1  If  there  is  one  precursor  transaction  Tj  of  7/  in  the  precursor  list  and  Tj  in  a  cluster 

Move  all  the  operation  of  T(to  the  cluster  of  T,; 

1 .4.1.2  Else 

1  .4.  1 .2. 1  If  all  the  precursors  are  in  temp_set 

Create  a  new  cluster; 

Move  all  operations  of  the  precursor  transactions  in  temp_set  to  the  cluster; 

Move  all  operations  of  Tj  to  the  cluster; 

1.4. 1.2.1  Else 

Merge  the  clusters  (if  there  are  more  than  one)  of  the  precursors  to  one  cluster; 

Move  all  operations  of  the  precursor  transactions  in  the  temp_set  (if  any)  to  the  same  cluster; 

Move  all  the  operation  of  T{  from  the  temp_set  to  the  same  cluster; 

2.  Delete  the  precursor  list  of  Tj. 

Correctness  of  this  algorithm  is  similar  to  that  of  the  clustering  algorithm  and,  therefore,  omitted  here. 

5.  Performance  Analysis 

We  developed  a  simulation  model  and  collected  several  sets  of  data  to  compare  the  number  of  transactions 
to  be  scanned  and  the  time  required  to  read  these  transactions  from  the  secondary  storage.  The  results  obtained  are 
highly  promising.  Our  model  clearly  outperforms  the  traditional  methods  in  all  cases.  Next,  we  present  a  brief 
description  of  the  simulation  model. 

In  our  simulation,  transactions  access  data  items  randomly.  The  operation  (whether  read,  or  write,  or 
read/write)  is  also  randomly  chosen.  The  simulation  program  keeps  a  read  and  a  write  set  for  each  transaction.  It 
maintains  a  data  structure  called  last_write  for  each  data  item  to  store  the  most  recent  write  operation  performed  on 
them.  When  there  is  a  read  operation  on  any  data  item,  the  last_write  table  is  checked  to  see  which  transaction 
wrote  the  item  last.  This  establishes  the  dependency  between  the  transaction  (precursor)  which  wrote  the  data  item 


8-9 


last  and  the  transaction  (dependent)  reading  the  data  item.  Once  the  dependency  is  established,  the  ids  of  the 
precursor  and  the  dependent  transaction  are  written  to  a  cluster.  Cluster  information  is  maintained  by  a  data 
structure  called  cluster_list. 

We  ran  our  simulation  on  two  varying  parameters.  First,  the  maximum  number  of  data  items  accessed  by 
the  transactions  was  changed  and  the  total  number  of  available  data  items  was  kept  fixed.  Then  we  kept  the 
maximum  number  of  data  items  accessed  by  each  transaction  and  ran  our  simulation  with  changing  total  number  of 
available  data  items.  We  ran  these  simulations  for  two  different  attackers  {Tjq  and  T20)  for  the  two  changing 
parameters  discussed  above.  Then  for  each  attacker,  we  plotted  two  graphs  comparing  the  number  of  transactions  to 
be  scanned  for  recovery  and  the  total  page  I/O  time  to  demonstrate  the  performance  of  our  model  and  the  log  based 
model.  Table  1  lists  the  system  dependent  fixed  parameters. 


Variable 

Description 

Value 

SQ 

Space  taken  by  the  commit  operation  of  a  transaction  in  the  log 

12  bytes 

SSj 

Space  taken  by  the  start  operation  of  a  transaction  in  the  log 

12  bytes 

SR, 

Space  taken  by  a  read  operation  of  a  transaction  in  the  log 

40  bytes 

SWj 

Space  taken  by  a  write  operation  of  a  transaction  in  the  log 

60  bytes 

PS 

Page  Size 

2KB 

PT 

Page  I/O  Time 

15  milliseconds 

Table  1:  System  Dependent  Parameters 


For  the  log  based  recovery  method,  the  recovery  process  begins  by  scanning  all  operations  of  transactions 
from  the  point  of  attack  to  the  end  of  the  log.  So  to  calculate  the  total  page  I/O  for  this  model  we  calculated  all  the 
read,  write,  start  and  commit  operations  of  transactions  starting  from  the  attacking  transaction  to  the  last  transaction 
in  the  log.  By  using  the  space  occupied  by  each  of  these  operations  as  listed  in  Table  1,  we  calculated  the  size  of  the 
part  of  the  log  that  needs  to  be  scanned  and  hence  the  total  page  I/O  time.  The  following  example  explains  our 
calculation. 


Example :  Assume  that  the  total  number  of  transactions  in  the  log  is  30  and  the  attacker  is  the  10th  transaction.  So  we 
need  to  scan  21  transactions  in  the  traditional  method.  If  these  21  transactions  have  a  total  of  150  read  operations 
and  200  write  operations,  we  compute  the  total  page  I/O  time  for  the  traditional  method  as  follows. 

Total  space  for  read  operations:  150  *  SR^  =  6000  bytes 
Total  space  for  write  operations:  200  *  SW(  =  12000  bytes 
Total  space  for  commit  operations:  21  *  SQ  =  252  bytes 
Total  space  start  operations:  21  *  SQ  -  252  bytes 
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Total  Space  needed  to  be  scanned:  6000  +  12000  +  252  +  252  =  18504  bytes 
Number  of  pages  needed:  [total  space  /  PS]  =10 
Total  page  I/O  time:  10  *  PT  =  150  milliseconds 

Above  mentioned  steps  have  been  used  to  calculate  the  total  page  I/O  or  Log  Access  Time  for  the 
traditional  log  based  methods.  To  calculate  the  total  page  I/O  for  our  method,  we  used  the  same  steps  but  the 
number  of  transactions  to  be  scanned  is  determined  from  the  appropriate  cluster. 

In  figures  2  through  5,  the  total  number  of  transactions  and  the  total  number  of  data  items  are  kept  constant 
whereas  the  maximum  number  of  data  items,  which  can  be  accessed  by  each  transaction,  has  been  varied.  In  figures 
1  and  2  the  10th  transaction  is  assumed  to  be  the  attacker.  In  figures  3  and  4  the  attacker  is  chosen  to  be  the  7th 
transaction. 


Figure  2:  Total  Transactions  to  be  Scanned  When  Attacker  is  T10 


Figure  3:  Log  Access  Time  Comparison  When  Attacker  is  T10 
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Figure  5:  Log  Access  Time  Comparison  When  Attacker  is  T*>0 


In  figures  6  through  9,  we  have  kept  total  number  of  transaction  and  max  number  of  data  items  accessed  fixed. 
We  plotted  the  graphs  with  varying  total  number  of  data  items  available.  The  attacker  is  chosen  to  be  the  10th 
transaction  in  figures  6  and  7,  and  as  the  20th  transaction  in  figures  8  and  9. 


Figure  6:  Total  Transactions  to  be  Scanned  When  Attacker  is  T10 
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Figure  7:  Log  Access  Time  Comparison  When  Attacker  is  T 


6.  Conclusion 

Every  computer  system  that  is  networked  is  vulnerable  to  information  attacks.  While  retaliation  to  these 
attacks  is  virtually  impossible,  immediate  attention  must  be  paid  for  database  survivability.  Quick  and  efficient 
recovery  of  the  system  is  vital  for  an  organization  whose  information  resources  have  been  attacked.  In  this  research, 
we  have  presented  two  algorithms  that  cluster  the  log  based  on  transaction  dependency.  The  first  algorithm  divides 
an  already  existing  log  into  several  clusters.  This  is  necessary  to  remodel  conventional  database  management 
systems  to  adapt  to  log  clustering  method.  The  second  algorithm  uses  already  clustered  log  to  place  committed 
transactions  in  appropriate  cluster.  We  expect  a  background  process  to  perform  this  activity  for  improved 
efficiency.  We  have  used  simulation  to  verify  the  performance  of  our  method.  Several  graphs  are  presented  to 
depict  the  results,  which  show  significant  improvement  in  damage  assessment  and  recovery  time  using  our  model. 
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Abstract 

Detection  of  inconsistency  of  frequency  databases  (collections  of  contingency  tables)  is  an 
NP-complete  problem.  As  has  been  demonstrated  for  other  NP-complete  problems,  it  is  possi¬ 
ble  to  identify  the  hardest  instances  of  the  consistency  detection  problem  in  terms  of  an  order 
parameter  analogous  to  those  for  phase  transitions  in  physical  systems.  In  addition  to  develop¬ 
ing  an  order  parameter,  this  report  presents  methods  for  utilizing  inconsistent  databases  in  deci¬ 
sion  making. 
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COMPLEXITY  OF  DETECTING  AND  CONTEXT-DRIVEN  METHODS 
FOR  RESOLVING  DATABASE  INCONSISTENCY 

Michael  A.  Pittarelli 


I.  Introduction 

Much  work  has  been  done  in  recent  years  on  the  problem  of  identifying  hard  instances  of 
constraint  satisfaction  problems  [1].  Most  of  these  problems  are  NP-complete  [2],  Although 
such  problems  (probably)  require  exponential  time  in  the  worst  case,  in  practice,  many 
instances  can  be  solved  reasonably  quickly. 

The  research  on  locating  hard  instances  of  these  problems  has  focussed  on  identifying  crit¬ 
ical  parameters  in  the  description  of  an  instance.  For  example,  for  randomly  generated  instances 
of  3-SAT,  the  difficult  cases  are  concentrated  around  a  particular  ratio  of  clauses  to  variables 
[3].  For  the  problem  of  determining  whether  a  partially  specified  latin  square  (operation  table 
for  a  quasigroup)  can  be  completed,  there  is  a  value  for  the  percentage  of  entries  that  are  fixed 
in  advance  around  which  the  difficult  cases  are  clustered  [4],  There  tends  to  be  an  abrupt 
change  in  the  performance  of  algorithms  as  one  goes  from  instances  characterized  by  values 
less  than  the  critical  value  to  instances  with  greater  values.  This  behavior  is  analogous  to  a 
phase  transition  in  a  physical  system. 

The  type  of  problem  dealt  with  in  this  report  is  that  of  determining  whether  a  collection  of 
partially  specified  tables  of  frequency  distributions  (a  frequency  database)  is  consistent.  The 
tables  below  represent  a  (completely  specified)  frequency  database  that  is  consistent: 


Gender 

Dog_Owner 

/(•) 

Dog_Owner 

SUV_Owner 

f 

Male 

Yes 

20 

Yes 

Yes 

12 

Male 

No 

10 

Yes 

No 

38 

Female 

Yes 

30 

No 

Yes 

4 

Female 

No 

40 

No 

No 

46 

The  database  is  consistent  because  the  two  tables  agree  on 

the  frequencies  for  the  shared 

attribute  "Dog_Owner";  it  can  be  inferred  from  either  table  that  there  are  50  dog  owners  in  the 

population  corresponding  to  the  database. 

A  completely  specified  database  is  a 

special  case  of  a  partially  specified  database.  For  a 

partially  specified  database,  0  or  more  of  the  frequencies  are  missing.  For  example: 

Gender 

Dog_Owner 

/(•) 

Dog_Owner 

SUV_Owner 

f 

Male 

Yes 

? 

Yes 

Yes 

12 

Male 

No 

10 

Yes 

No 

? 

Female 

Yes 

? 

No 

Yes 

4 

Female 

No 

? 

No 

No 

46 

The  database  is  consistent,  since  it  can  be  completed  in  such  a  way  that  there  exists  a  frequency 
distribution  over  the  three  attributes  Gender,  Dog_Owner  and  SUV_Owner  whose  marginals 
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over  the  pairs  of  attributes  {Gender,  Dog_Owner},  {Dog_Owner,  SUV_Owner}  coincide  with 
those  given  in  the  partially  specified  tables.  (The  concept  of  consistency  is  more  thoroughly  dis¬ 
cussed  in  Section  HI,  which  also  contains  formal  definitions  of  various  other  database  con¬ 
cepts.) 

Section  II  of  this  report  details  the  work  done  on  attempting  to  develop  a  measure  (order 
parameter)  that  would  allow  prediction,  based  on  parameters  describing  the  problem  instance, 
of  whether  a  partially  specified  database  instance  is  likely  to  be  consistent  and  also  of  the  rela¬ 
tive  difficulty  of  determining  whether  the  instance  is  consistent. 

An  equally  important  component  of  the  research  conducted  deals  with  the  use  to  which  a 
database  of  marginal  frequencies  could  be  put.  Statistical  frequencies  are  normally  used  as  the 
basis  for  estimating  probabilities.  Probabilities,  in  turn,  are  typically  used  as  the  basis  for  deci¬ 
sion  making.  In  Section  III,  methods  for  decision  making  using  possibly  inconsistent  frequency 
tables  will  be  discussed. 

II.  Experiments 

We  conducted  experiments  to  locate  difficult  instances  of  the  problem  of  determining 
whether  partially  specified  frequency  database  tables  are  consistent.  The  experiments  involved 
binary  attributes  and  database  schemes  of  a  very  restricted  form,  as  described  below.  The  gen¬ 
eral  framework,  for  the  case  of  3  attributes,  is  as  follows: 


Aj 

^2 

/. 

A-2 

A3 

Si 

0 

0 

*8 

0 

0 

*12 

0 

1 

*9 

0 

i 

*13 

1 

0 

*10 

1 

0 

*14 

1 

1 

*11 

1 

i 

*15 

A)  A2  A3  /q 

0  0  0  jt0 

0  0  1  JC, 

0  10  jc2 

Oil  jc3 
10  0  jc4 

10  1  jc5 

1  1  0  jc6 

1  1  1  x1 

The  fi  are  frequency  distributions.  The  constraints  are: 

xt  >  0 

x0  +  xx  +  ■  ■  ■  +  x6  +  x-j  =  TV 

*8  =  *0  +  *1 
*9  =  *2  +  X3 

*14  =  *2  +  *6 

*15  =  *3  +  X7 

11  15 

Note  that  these  equations  imply  the  constraints:  £  x,  =  N,  £  */  =  N. 

i=8  i=12 
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Zero  or  more  of  the  marginal  probabilities  x8, . . . ,  x15  have  values  preassigned  to  them. 
(The  case  in  which  all  of  the  marginals  have  values  assigned  to  them  was  proved  in  1977  to  be 
NP-complete  [5].)  None  of  the  joint  probabilities  have  values  preassigned.  The  problem  is  to 
determine  whether  values  satisfying  the  constraints  can  be  found  for  the  remaining  variables.  If 
so,  then  the  partially  specified  marginal  frequency  tables  are  consistent.  Consistency  can  also  be 
defined  for  partially  specified  tables  in  the  absence  of  a  specified  sum  N.  In  that  case,  the 
tables  are  consistent  if  there  exists  at  least  one  fully  specified  joint  table  (involving  each  of  the 
attributes  in  the  marginal  tables)  whose  marginals  agree  with  those  given  in  the  marginal  tables. 
(Inconsistency  will  be  discussed  more  thoroughly  in  Section  XXXX.  Section  XXXXXXXXX 
discusses  methods  for  utilizing  inconsistent  frequency  tables.) 

The  number  of  variables  can  be  reduced.  The  marginal  tables  can  be  reexpressed  as: 


A, 

a2 

f\ 

a2 

a3 

h 

0 

0 

x0  +  x} 

0 

0 

X0  +  X4 

0 

1 

x2  +  x3 

0 

1 

X\  +  JC5 

1 

0 

X4  +  X5 

1 

0 

*2  + *6 

1 

1 

*6  + *7 

1 

1 

X3  +  X-j 

Notice  that  in  each  subdistribution,  each  of  the  x,  appears  exactly  once.  (This  is  not  the  case  for 
more  general  database  schemes.)  Also,  for  database  schemes  like  this  one,  of  the  form 

{ {^1  >  ^2,  ■  ■  ■,  A„_j } , { A2,  A3,  ■  ■,  Aw } } , 

each  sum 

Xj  +  Xj 

is  of  one  of  two  simple  forms: 

a)  ie{0,  1, 2,  ■  •  2"-1-!}  and  j  =  i  +  2n~l 

b)  ie {0,  2, 4,  •  •  2"-2}  and  j  =  i+  1. 

The  maximum  number  of  marginal  constraints  is  always  2",  which  equals  the  number  of 
joint  tuples  (i.e.,  the  number  of  variables). 

In  the  latin  square  completion  problem,  the  parameters  are  the  number  of  rows  and 
columns  and  the  number  of  preassigned  entries.  The  number  of  different  values  for  the  entries 
is  equal  to  the  number  of  rows.  For  the  3-SAT  problem,  the  parameters  that  can  be  controlled 
are  the  number  of  propositional  variables  and  the  number  of  clauses.  For  this  problem,  there  are 
three  parameters  (vs.  two):  the  number  of  attributes,  the  number  of  preassigned  marginal  fre¬ 
quencies,  and  the  value  of  N. 

In  early  experiments,  it  was  attempted  to  control  for  the  interaction  of  N  with  the  other 
two  parameters  by  keeping  the  sum  of  the  number  of  partitions  of  N  into  2”  parts  constant.  For 
the  numbers  of  attributes  in  the  experiments  conducted  (n  =  4, 5, 6, 7),  N  =  32  kept  this  quantity 
approximately  constant.  (Note:  A  more  "realistic"  value  might  have  been  something  like  5x2" 
-  the  recommended  minimum  number  of  observations  for  the  applicability  of  certain  statistical 
procedures.  However,  the  set  of  possible  values  {0,1  for  the  variables  would  have  been 
too  large  to  conduct  the  experiments.  For  SAT  problems,  the  variables  have  only  two  possible 
values,  vs.  iV+1.  For  the  latin  square  problem,  the  number  of  values  is  equal  to  the  number  of 
rows,  but  results  have  been  reported  for  squares  with  at  most  20  rows.) 
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The  difficult  instances  of  latin  square  completion  problems  occur  when  approximately 
43%  of  the  elements  are  preassigned,  regardless  of  the  number  of  rows.  Similarly,  for  3-SAT 
problems,  the  hard  cases  arise  when  the  ratio  of  clauses  to  variables  is  approximately  4.3, 
regardless  of  the  number  of  variables.  These  problems  lack  an  analogue  of  the  global  constraint 
that  the  sum  of  all  of  the  variables  equals  N. 

This  combination  of  a  relatively  weak  constraint  involving  all  of  the  variables  and  con¬ 
straints  involving  pairs  of  variables  (values  for  specified  marginal  frequencies,  equal  to  the  sum 
of  a  pair  of  joint  frequencies)  complicated  the  search  for  a  phase  transition,  i.e.,  a  way  of  identi¬ 
fying,  independently  of  the  number  of  attributes,  the  point  at  which  determining  whether  or  not 
a  pair  of  frequency  tables  is  consistent  becomes  especially  difficult. 

This  and  other  attempts  at  controlling  for  factors  other  than  the  percentage  of  preassigned 
marginals  (thus  allowing  its  use  as  an  order  parameter)  were  unsuccessful. 

Another  fundamental  problem  involved  the  fact  that  the  number  of  joint  frequencies  is  an 
exponential  function  of  the  number  of  attributes  for  a  given  problem  instance  (2\  where  n  is 
the  number  of  attributes).  This,  combined  with  the  desire  for  a  realistic  average  marginal  value, 
severely  limits  the  number  of  problem  ensembles  that  can  be  generated. 

A  solution  (suggested  by  Bart  Selman)  to  the  problem  of  explosive  growth  of  the  state 
space  as  the  number  of  attributes  is  increased  is  to  keep  the  number  of  attributes  fixed  at  3,  but 
vary  the  domain  size  of  one  of  them  (a  "superattribute").  When  the  domain  size  of  the  superat¬ 
tribute  is  an  integer  power  ( i )  of  2,  the  problem  is  isomorphic  to  a  multiattribute  problem  with  i 
attributes.  For  the  original  problem  instances,  the  possible  numbers  of  joint  frequencies  are: 

8,16,32,64,128,256,512,... 

For  the  "superattribute"  version,  the  possible  numbers  of  joint  frequences  are  multiples  of  4: 

8,12,16,20,24,28,32,... 

With  database  scheme 


{ Mi,  ^2}'  {^2>  ^3} }> 

where  A,  and  A3  are  binary  and  A2  is  d- ary,  the  joint  tuples  are  partitioned  into  d  equivalence 
classes,  where  two  joint  tuples  are  in  the  same  class  if  they  agree  on  the  value  for  (have  the 
same  projection  onto)  the  superattribute  (A2).  There  is  a  total  of  4 d  marginal  frequencies.  Each 
marginal  frequency  is  the  sum  of  (constrains)  two  joint  frequencies.  (Similarly,  the  marginal 
tuples  are  partitioned  into  d  equivalence  classes).  Again,  there  are  two  marginal  frequency 
tables:  fx  over  A,  and  A2,  and  f2  over  A2  and  A3.  The  structure  of  the  constraints  may  be  repre¬ 
sented  as  an  undirected  graph  whose  edges  represent  marginal  frequencies  and  whose  nodes 
represent  joint  frequencies.  Two  nodes  are  connected  by  an  edge  if  their  sum  is  the  value  of  the 
marginal  frequency  that  the  edge  represents.  For  example,  when  d= 2,  the  graph  is: 
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x0  is  the  frequency  of  tuple  AXA2A3  =  000,  xx  is  the  frequency  of  AXA2A2  =  001,  ...,  and  xn  is 
the  frequency  of  Ax  A2A3  =  111. 

The  total  number  of  marginal  frequencies  is  4 d  (as  is  the  total  number  of  joint 
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frequencies).  In  the  experiments  that  were  performed,  the  number  of  randomly  generated 
marginal  frequencies  varied,  and  never  reached  more  than  about  30%  of  4 d.  Thus,  the  problems 
are  analogous  to  latin  square  (quasigroup)  completion  problems. 

Any  of  the  d  graphs  for  the  equivalence  classes  (A2  =  0,  A2  =  1, . .  • ,  A2  =  d)  may  have  0 
to  4  edges.  If  there  are  0  edges,  the  joint  frequencies  are  constrained  only  by  the  global  con¬ 
straints  (nonnegativity,  sum  =  N).  If  there  is  one  edge,  then  exactly  two  joint  frequencies  are 
constrained  by  one  marginal  frequency.  If  there  are  two  edges  for  a  single  equivalence  class, 
then  there  are  two  possibilities: 

(0  The  marginals  are  from  the  same  marginal  distribution,  in  which  case  two  disjoint  sets  of 
two  joint  frequencies  are  constrained.  For  example: 

X0  +  *1  =  m\ 
x4  +  x5  =  m2 

(ii)  The  marginals  are  from  different  marginal  distributions,  in  which  case  three  joint  frequen¬ 
cies  are  constrained.  For  example: 

x0  +  Xi=  mi 

+  x5  =  m2 

The  difference  in  the  degree  to  which  the  overall  solution  is  constrained  in  the  two  cases 
depends  on  N  and  on  the  values  of  the  marginals.  Assume  that 

m{  =  m2  =  k. 

For  the  subproblem  in  which  the  joint  frequencies  constrained  by  the  marginals  are  considered 
in  isolation  from  any  other  constraint  (e.g.,  that  all  the  joint  frequencies  must  sum  to  N),  the 
number  of  solutions  for  case  (i)  is  (k  +  l)2  However,  as  a  fraction  of  size  of  the  state  space  for 
this  subproblem,  this  is 

(i k  +  1  )2/(N  +  l)4 

For  case  (ii),  there  are  only  k  +  1  solutions;  the  value  of  any  one  of  the  three  joint  frequencies 
determines  the  values  of  the  other  two.  So,  the  fraction  of  the  elements  of  the  (sub)state  space 
that  are  solutions,  in  case  (ii),  is: 

(k  +  1  )/(N  +  l)3 

These  fractions  are  plotted  for  N  =  256  and  k  =  N/2d,  the  average  value  of  a  marginal  if  each 
marginal  is  assigned  a  value,  in  Figure  1 . 

With  3  or  4  marginals  in  one  equivalence  class,  all  4  joint  frequencies  are  constrained. 
Again  assuming  that  each  marginal  equals  some  constant  k,  the  fraction  of  solutions  in  the 
(sub)state  space  is  (c  +  1  )/(N  +  l)4,  in  either  case. 

Experiments  were  conducted  for  domain  sizes  of  10,  12,  14,  16  and  18  for  the  superat¬ 
tribute.  For  each  domain  size,  the  number  of  marginals  randomly  assigned  values  was  varied 
from  about  2%  of  the  total  number  of  marginals  to  about  40%.  (Although  the  percentage  of 
marginals  assigned  is  not  an  order  parameter  for  this  problem,  as  discussed  below,  for  each 
domain  size  the  transition  from  nearly  all  consistent  to  nearly  all  inconsistent  occurred  well 
before  40%  of  the  marginals  were  assigned  values.) 

In  Figure  2,  the  25th  percentile  of  the  number  of  backtracks  required  to  determine  whether 
or  not  a  database  is  consistent  is  plotted  against  the  percentage  of  marginals  selected  and  given 
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values.  The  procedure  used  to  determine  consistency  was  a  reverse  Brelaz  heuristic,  imple¬ 
mented  in  C++,  using  ILOG  Solver  libraries  [6,  7].  The  program  was  adapted  from  programs 
developed  by  Carla  Gomes  for  the  quasigroup  completion  problem  [4].  (Figures  3  and  4, 
respectively,  plot  backtracks  for  domain  sizes  12  and  16  separately.)  As  Figure  2  illustrates,  the 
percentage  of  marginals  selected  at  which  the  number  of  backtracks  is  maximized  decreases  as 
the  domain  size  increases.  (In  the  experiments  conducted,  from  20.0%  for  domain  size  =  10  to 
18.1%  for  domain  size  =  18.)  Therefore,  the  percentage  of  marginals  selected  does  not  appear 
to  be  an  order  parameter  for  the  database  consistency  problem. 

Following  the  unsuccessful  attempts  to  keep  invariant  the  degree  to  which  a  problem 
ensemble  is  constrained,  by  keeping  constant  the  number  of  solutions  in  the  case  in  which  none 
of  the  marginals  are  preassigned,  we  became  aware  of  the  work  of  researchers  at  the  University 
of  Leeds  and  Strathclyde  University  on  characterizing  the  "constrainedness"  of  ensembles  of 
combinatorial  search  problems.  Gent  et  al.  [8]  measure  the  constrainedness  of  an  ensemble  of 
problems  as 

1  -  log, (<  Sol  >)/log2(ISI) 

where  <  Sol  >  is  the  expected  number  of  solutions  for  a  given  problem  in  the  ensemble  and  S  is 
the  state  space  for  the  problem.  Good  results  (and  agreement  with  measures  developed  indepen¬ 
dently)  are  shown  for  this  measure  for  a  variety  of  combinatorial  search  problems:  satisfiability, 
graph  coloring,  general  constraint  satisfaction,  and  number  partitioning. 

For  the  superattribute  version  of  the  database  consistency  problem,  the  size  of  the  state 
space  is 

(N  +  1  fd, 

where  N  is  the  specified  sum  (of  the  joint  frequencies  and  of  the  marginal  frequencies,  for  both 
marginal  frequency  tables)  and  d  is  the  domain  size  for  the  superattribute. 

The  expected  number  of  solutions,  <  Sol  >,  can  be  calculated  as  the  size  of  the  state  space 
times  the  fraction  of  solutions  determined  by  the  parameters  for  a  problem  ensemble  (domain 
size,  sum,  number  of  preassigned  marginals).  For  this  problem,  there  is  a  global  constraint  (sum 
of  joint  frequencies  =  N )  and  p  "local"  constraints,  one  for  each  of  the  p  preassigned 
marginals.  The  fraction  of  the  state  space  satisfying  the  global  constraint  is 

C(4d  +  N  -  1,  N) 

(N  +  l)4rf  ' 

Considered  independently,  each  of  the  p  preassigned  marginal  values  constrains  two  joint  fre¬ 
quencies.  Let  nij  denote  the  (expected)  value  of  the  yth  marginal.  Then  for  the  (sub)state  space 
consisting  of  pairs  of  values  for  the  affected  joint  frequencies,  the  fraction  satisfying  yth 
marginal  constraint  is 

nij  +  1 

W+ 1)2 ' 

On  the  assumption  of  independence  of  the  marginal  constraints,  then, 


..  p  m,-  +  1 

<  Sol  >  =  (N  +  1)4<*  x  n  . —  x 


j= i  (N+\)2 


C(4d  +  N  -  1,  JV) 

(AT  +  l)4rf 


p  nij  +  1 

n  — ^ — 

;=1  (A  +  l)2 


x  C(4d  +  N  -  1,  N). 


Gent  et  al.  refer  to  their  constrainedness  measure  as  kappa  (k),  and  we  will  refer  to  various 
refinements  of  this  measure  for  this  problem  as  kappa-0,  kappa- 1,  etc.  Kappa-0  will  refer  to  the 
version  in  which:  the  marginal  constraints  are  assumed  independent  of  each  other;  the 
(expected)  marginal  values  are  assumed  equal  to  each  other;  and  the  expected  marginal  value 
equals  N/2d.  Figure  5  plots  backtracks  against  kappa-0  for  domain  sizes  10,  12,  14,  16,  and  18. 
The  "drifting  to  the  left"  of  the  peaks  evident  in  Figure  2  does  not  occur  in  Figure  5.  However, 
there  is  clearly  room  for  improvement. 

We  will  call  "kappa- 1"  the  refinement  of  kappa-0  in  which  the  expected  marginal  is  calcu¬ 
lated  taking  into  account  the  manner  in  which  the  marginals  are  generated  experimentally. 
Kappa-0  assumes  that  the  expected  marginal  value  equals  the  table  sum  (N)  divided  by  the 
number  of  marginal  frequencies  per  table  (2d).  This  assumption  is  correct  if  each  marginal  is 
preassigned.  However,  the  percentage  of  preassigned  marginals  never  exceeded  35.7%  in  the 
experiments  conducted.  Furthermore,  in  the  experiments,  each  table  was  allocated  N  units  to 
distribute  (although  not  necessarily  to  exhaust)  among  the  marginal  frequencies  selected  from 
that  table.  For  each  marginal  value  randomly  generated  (with  uniform  probability  over  the  inte¬ 
gers  from  0  to  the  number  of  allocated  units  remaining),  the  remaining  units  were  reduced  by 
that  value.  Thus,  the  expected  value  of  the  first  marginal  for  a  marginal  table  would  be  N/2;  the 
expected  value  for  the  ith  marginal  from  a  table  would  be  N/21.  However,  the  program  used  for 
these  experiments  limited  the  marginal  values  to  8  times  what  the  average  marginal  value 
would  be  if  the  marginal  tables  were  completely  specified:  4N/d.  Without  this  restriction  on 
the  range  of  marginal  values,  the  problem  ensembles  generated  tended  not  to  show  very  dra¬ 
matic  transitions  from  easy  to  hard  to  easy  regimes.  Figure  6  plots  backtracks  against  kappa- 1, 
where  values  for  kappa- 1  are  based  on  experimentally  observed  average  marginal  values. 

The  value  of  kappa- 1  for  a  particular  (N,  d,  p )  triple  is  lower  than  that  of  kappa-0.  The 
empirically  observed  average  marginal  value  used  in  the  calculation  of  kappa- 1  is  higher  than 
the  estimate  used  by  kappa-0;  thus  the  contribution  of  each  marginal  to  <  Sol  >  is  greater,  in 
kappa- 1:  m  >  m  implies 

m  +  1  ^  m  +  1 

(N  +  l)2  >  (N  +  l)2 


Furthermore,  the  values  of  kappa- 1  for  the  hardest  cases  for  the  two  largest  domain  sizes  (16 
and  18)  are  equal  (to  3  decimal  places).  However,  the  fact  that  the  numbers  of  preplaced 
marginals  are  integers,  and  are  somewhat  small,  reduces  the  significance  of  this  result. 

Kappa- 1  still  assumes  that  the  marginals  (for  a  given  N,  d  and  p )  have  the  same  expected 
values.  Kappa-2  is  calculated  using  separate  expected  values  for  each  of  the  preplaced 
marginals.  Although  the  variations  are  in  some  cases  quite  extreme  (e.g.,  6.8  for  the  1st 
marginal,  with  N  =  56  and  d  =  14,  vs.  2.1  for  the  20th  marginal)  the  effect  on  the 
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constrainedness  measure  is  not  very  dramatic.  The  values  for  the  hardest  cases  for  the  three 
largest  domain  sizes  (14,  16  and  18)  are  in  slightly  better  agreement.  In  general,  the  value  of 
kappa-2  for  any  particular  N,  d  and  p  is  slightly  smaller  than  the  corresponding  value  for 
kappa- 1.  This  is  due  to  the  fact  that 

flintj  +  l)<(E(m.  +  \)lnf 
J=l  j= i 

Backtracks  are  plotted  against  kappa-2  in  Figure  7. 

The  peak  effort  for  domain  sizes  10  and  12  occurs  at  lower  values  than  for  sizes  14,  16  and 
18,  for  each  version  of  kappa  (-0,  -1  and  -2).  For  kappa-2,  the  peaks  occur  at  0.933  and  0.931 
for  d  =  10  and  d  =  12,  respectively,  vs.  0.959,  0.957  and  0.956,  respectively,  for  d=  14,  d  =  16 
and  d  -  18. 

A  plausible  explanation  is  that  independence  of  the  marginal  constraints,  which  is  assumed 
by  each  of  these  measures,  is  more  likely  for  the  smaller  domain  sizes,  for  the  number  of  preas¬ 
signed  marginals  generating  the  most  backtracks.  As  discussed  previously,  for  a  problem 
instance  with  p  preassigned  marginals  and  domain  size  d,  the  p  constraints  are  distributed 
among  d  equivalence  classes.  A  sufficient  condition  for  independence  of  the  p  marginals  is  that 
they  are  distributed  among  the  d  equivalence  classes  in  such  a  way  that  no  class  is  assigned 
more  than  one  marginal.  In  other  words,  the  mappings  from  marginals  to  equivalence  classes  is 
1-1.  The  fraction  of  1-1  mappings  from  marginals  to  classes, 

d\/(d-  p)\ 

~dP  ’ 

is  quite  low  (at  the  point  of  peak  backtracks)  for  all  the  problem  instances  studied  but  does 
increase  as  the  domain  size  for  the  superattribute  decreases: 

d=  18,  p-  13:  0.003 
d  =  16,  p=  12:  0.003 
d  =  14,  p  =  11:  0.004 
d  =  12,  p  =  9\  0.015 
d  =  10,  p  =  8:  0.018. 

Furthermore,  there  is  a  steep  increase  in  this  value  as  d  decreases  from  14  to  12. 

Therefore,  the  expected  number  of  solutions  is  underestimated  (and  the  degree  of  con¬ 
straint  is  overestimated)  to  a  greater  extent  for  larger  values  of  of  d.  For  example,  with  two 
marginals  in  a  particular  class,  the  probability  is  1/3  that  the  two  constraints  are  independent. 
For  d  =  10,  p  -  8  and  the  observed  average  marginal  value  of  7.5,  the  solution  fraction  calcu¬ 
lated  by  our  measures,  which  assume  independence,  is  3  x  10-5.  However,  if  the  two  marginals 
are  not  independent  (the  probability  of  which  is  2/3),  the  solution  fraction  calculated  should  be 
1  x  10-4.  For  d  =  18  and  p  =  13,  the  fractions  are  3  x  10-6  and  2  x  10-5.  Although  each  dif¬ 
ference  is  a  single  order  of  magnitude,  the  ratio  is  twice  as  large  for  <7=18. 

Mappings  from  marginals  to  equivalence  classes  can  be  distinguished  by  the  number  of 
marginals  occupying  each  class,  where  the  number  in  each  class  is  an  integer  in  the  range  0  to 
4.  Call  an  assignment  of  an  integer  in  the  range  0  to  4  to  each  equivalence  class,  where  the  sum 
of  the  integers  is  p,  a  configuration.  Each  configuration  has  a  probability.  With  the  simplifying 
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assumption  that  each  marginal  (1st,  2nd, ...,  pth,  for  a  given  d,  p  and  N)  has  the  same  expected 
value,  m,  the  expected  number  of  solutions  can  be  calculated  for  each  configuration,  weighted 
by  the  corresponding  probability,  and  summed.  For  example,  when  d  =  2  and  p  =  3,  the  equiv¬ 
alence  classes  are: 
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The  number  of  configurations  is  20:  4  in  which  class  d  =  0  is  empty  and  class  d  =  1  has  3 
members,  in  which  case  the  marginals’  contribution  to  the  calculation  of  <  Sol  >  is 


m  +  1 
(N  +  l)4  ’ 

6  in  which  class  d  =  0  has  one  member  and  class  d  =  1  has  2  members,  where  the  contribution 
to  <  Sol  >  is 


1  (m  +  l)2  2 

3  X  (N  +  l)4  +  3  X 


(m  +  1) 

(N  +  l)3 


;  etc. 


I  have  not  implemented  a  version  of  constrainedness  incorporating  these  refinements;  I  hope  to 
work  on  this  problem  in  the  very  near  future  with  someone  with  expertise  in  physics  (e.g.,  Bart 
Selman). 

Gomes,  Selman  and  Crato  [9]  have  discovered  that  some  instances  of  combinatorial  search 
problems  have  extreme  variability  in  the  amount  of  computational  effort  required  to  solve  them 
using  different  heuristics,  or  for  stochastic  methods,  different  random  seeds.  For  some  of  these 
instances,  the  distributions  are  "heavy-tailed".  Roughly,  for  a  stochastic  algorithm,  the  running 
mean  of  the  measure  of  computing  effort  (time,  number  of  backtracks)  over  repeated  trials  with 
the  same  instance  does  not  converge.  No  instances  of  the  database  consistency  problem  were 
found  with  this  behavior,  despite  spending  hundreds  of  cpu  hours  in  the  search  for  such 
instances.  The  search  was  concentrated  on  instances  near  the  peak  number  of  backtracks  for  a 
given  domain  size.  Furthermore,  since  consistent  instances  were  much  more  likely  to  exhibit 
wide  variation  in  the  effort  required  to  solve  them,  the  search  was  eventually  limited  to  such 
cases. 

Eventually,  in  the  course  of  searching  for  heavy-tailed  problem  instances,  experiments 
were  devised  for  database  instances  with  a-cyclic  schemes  [10].  Such  schemes  are  well-known 
to  have  undesirable  properties,  for  example,  with  respect  to  efficient  join  processing.  (Cyclicity 
of  database  schemes  is  discussed  in  Section  III.D.) 

For  these  experiments,  there  were  n  binary  attributes  and  n  marginal  tables,  each  involving 
n  -  1  of  the  attributes.  Whereas  the  "constraint  graph"  for  a  "superattribute"  database  scheme 
with  domain  size  d  consists  of  d  disconnected  4-node,  4-edge  undirected  graphs  (see  above), 
the  constraint  graph  for  these  schemes  may  be  visualized  as  2n_3  cubes  (12  marginals/edges 
each)  plus  edges  connecting  each  vertex  with  2'1-4  vertices  in  other  cubes  (for  n  >  4;  for  n  =  3, 
the  graph  is  a  single  cube).  So,  for  n  =  4,  the  graph  may  be  laid  out  so  as  to  resemble  a  box 
kite;  for  n  =  5,  two  loosely  connected  box  kites;  etc.  Figure  8  plots  median  backtracks  against 
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the  percentage  of  marginals  preassigned,  and  figure  9  plots  backtracks  against  kappa- 1  for  6 
attributes  with  N  =  96,  and  7  attributes  with  N  =  128.  (Note  that  the  percentage  at  which  the 
backtracks  are  maximum  is  halved  when  the  number  of  attributes  is  doubled  and  that  the  per¬ 
centages  are  much  lower  than  for  the  superattribute  experiments.) 

Although  many  of  the  cases  generated  exhibited  wide  variation  in  number  of  backtracks 
when  repeated  with  a  randomized  algorithm,  none  exhibited  heavy  tails.  Figure  10  plots  the 
number  of  backtracks  required  for  a  solution  versus  trial  number,  for  200  repetitions  of  a  ran¬ 
domized  first-fail  heuristic  applied  to  a  particular  problem  instance  with  5  attributes  and  7 
marginals.  Figure  1 1  plots  the  running  mean  of  the  number  of  backtracks  versus  trial  number, 
and  is  clearly  converging. 

Figure  12  shows  a  screen  from  a  Java  applet  that  conducts  a  systematic  search  for  a  solu¬ 
tion  to  randomly  generated  or  user-determined  database  consistency  problem  instances,  for  a 
cyclic  scheme  with  3  binary  attributes.  The  URL  is: 

http://www.cs.sunyit.edu/faculty/mike/DB.html 
Additional  features  (stochastic  search,  restarts)  will  be  added  in  the  very  near  future. 

III.  Using  frequency  database  tables 
A.  Motivating  example 

Consider  a  scenario  involving  a  target  moving  within  a  3-dimensional  space.  Label  the 
dimensions  X,  Y,  and  Z.  We  can  observe  the  target’s  location  in  the  X  -  Y  plane  and  Y  -  Z 
plane,  but  not  the  X  -  Z  plane.  However,  we  are  interested  in  its  location  in  the  X  -  Z  plane 
(parallel  to  the  ground).  The  reason  for  our  interest  is  that  we  must  decide  where  in  the  X  -  Z 
plane  to  drop  a  large  object  (without  the  capability  of  tracking,  seeking,  etc.).  Ideally  the  object 
will  hit  the  target.  However,  if  we  miss,  we  want  to  avoid  destroying  valuable  resources  on  the 
ground. 

For  simplicity,  suppose  each  dimension  is  divided  into  2  regions,  left-right  or  up-down. 
Label  them  0  and  1:  X  =  0,  X  =  1,  Y  =  0,  etc.  (The  number  of  regions  can  be  made  arbitrarily 
greater  than  2  for  greater  realism.)  Each  plane  is  divided  into  4  quadrants:  X  =  0,  Y  =  0;  X  =  0, 
Y  =  1 ;  etc.  The  X  -  Y  -  Z  cube  is  divided  into  8  octants:  X  =  0,  Y  =  0,  Z  =  0;  etc.  We  will  use 
the  abbreviations  XY  =  01  for  X  =  0,  Y  =  1,  etc. 

Suppose  our  data  are  in  the  form  of  frequencies  of  location  recorded  by  two  sen¬ 
sors/observers,  one  monitoring  the  X  -  Y  plane  and  the  other  monitoring  the  Y  -  Z  plane.  The 
quadrant  that  the  target  is  (mostly)  occupying  is  recorded  for  both  of  the  two  planes  at  each  of 
N  equally  spaced  instants.  For  example,  77=100  and  at  time  instant  33,  the  target  is  mostly 
occupying  quadrant  XY  =  01  in  the  X  -  Y  plane  and  quadrant  YZ  =  1 1  in  the  Y  -  Z  plane. 

After  10,000  observations,  suppose  the  data  are: 
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x  Y  /,(.) 


Y  Z  f2(.) 


0  0  2000 

0  1  1000 

1  0  3000 

1  1  4000 


0  0  4500 

0  1  500 

1  0  1000 

1  1  4000 


Recall  that  our  interest  is  in  the  X  -  Z  plane,  which  cannot  be  observed  directly.  What  do 
the  available  data  tell  us  about  frequencies  in  the  X  -  Z  plane?  First,  note  that  each  of  the  8 
numbers  in  the  frequency  tables, 

/,  (XY  =  00)  =  2000,  MXY  =  01)  =  1000, ....  f2(YZ  =  11)  =  4000, 
is  a  marginal  frequency,  relative  to  some  unknown  frequency  distribution  /3  over  X,  Y  and  Z. 
For  example, 

2000  =  MXY  =  00)  =  MXYZ  =  000)  +  MXYZ  =  001). 

To  simplify  notation,  let  nt  stand  for  MXYZ  =  abc ),  where  i  is  the  decimal  representation  of 
the  bit  string  abc.  Then  the  frequency  distributions  /)  and  f2  determine  a  system  of  linear  inte¬ 
ger  equations  the  set  of  solutions  of  which  corresponds  to  the  set  of  all  the  frequency  distribu¬ 
tions  /3  over  X,  Y,  and  Z  whose  projections  onto  X  -  Y  and  Y  -  Z,  respectively,  are  the  distri¬ 
butions  /,  and  M 

n0  +  n:=  2000 
n2  +  «3  =  1000 
n4  +  n5  =  3000 
n6  +  n-j  =  4000 
n0  +  nA  =  4500 
n\  +  ns  -  500 
n2  +  n6=  1000 
n3  +  n2  =  4000 


(Note  that  the  first  4  and  second  4  equations  each  imply  the  constraint 

n0  +  ni  +  •  •  •  +  n2  =  10, 000.) 

If  there  existed  a  single  solution  to  this  system  of  equations,  call  it 

<n0*,  »!*,••■,  »7  *>> 

we  could  calculate  the  values  of  the  desired  frequencies  as  in  the  table  below: 
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X  z 


W> _ 

0  0  /3(XFZ  =  000)  +  /3(XKZ  =  010)  =  «o*  +  «2!,! 

0  1  fi(XYZ  =  001)  +  /3(XTZ  =  01 1)  =  *  +  n3  * 

1  0  h(XYZ=  \00)  +  MXYZ  =  110)  =  n4*  +  «6* 

1  1  MXYZ  =  101)  +  MXYZ  =  111)  =  n5*  +  n7* 

However,  there  are  many  possible  solutions.  Here  are  three: 


X 

Y 

z 

fH) 

fl  (•) 

/’(.) 

0 

0 

0 

2000 

1500 

1800 

0 

0 

1 

0 

500 

200 

0 

1 

0 

500 

0 

200 

0 

1 

1 

500 

1000 

800 

1 

0 

0 

2500 

3000 

2700 

1 

0 

1 

500 

0 

300 

1 

1 

0 

500 

1000 

800 

1 

1 

1 

3500 

3000 

3200 

Suppose  /3  is  the  actual  frequency  distribution  of  which  fx  and  f2  are  projections,  and 
that 

nx  =/3'(XKZ  =  001)  =  0 

because  there  is  a  mountain  occupying  most  of  that  octant.  /32  is  one  of  the  solutions  /3'  of  the 
system  of  equations  whose  value  for  nx  is  maximum.  Each  /]  has  a  corresponding  relative  fre¬ 
quency  distribution  rf3',  calculated  as 

rfi(x)  =  fi(x)/ 10,000. 

The  entropy  of  a  relative  frequency  distribution  rf  is  calculated  as 

H(rf)  =  -'Lxrf(x)xln(rf(x)). 

Distribution  /3  is  the  element  of  the  solution  set  whose  corresponding  relative  frequency  distri¬ 
bution  has  highest  entropy.  Roughly,  this  means  that  it  is  the  distribution  that  is  closest  (on  vari¬ 
ous  metrics)  to  the  uniform  distribution.  (If  the  uniform  distribution  were  a  member  of  the  solu¬ 
tion  set,  it  would  be  the  maximum  entropy  solution.)  There  are  a  number  of  arguments  for 
selecting  the  maximum  entropy  distribution  from  the  set  of  distributions  consistent  with  a  given 
collection  of  constraints  [11,  12].  There  are  also  counterarguments  [13,  14]. 

The  projections  of  /3 ,  /32  and  f\  onto  X  and  Z  are  named,  respectively,  /4 ,  /42  and  /4  in 
the  table  below: 
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X 

z 

/„'(.) 

/42C) 

/40) 

0 

0 

2500 

1500 

2000 

0 

1 

500 

1500 

1000 

1 

0 

3000 

4000 

3500 

1 

1 

4000 

3000 

3500 

To  add  to  the  confusion,  note  that  there  is  a  second  set  of  distributions  that  it  would  be  rea¬ 
sonable  to  construct  from  the  original  data.  We  may  choose,  in  effect,  to  ignore  the  information 
regarding  the  Y  dimension,  and  project  /,  onto  X  and  f2  onto  Z: 

X  /5(.)  Z  /6(.) 

0  3000  0  5500 

1  7000  1  4500 

Distributions  f5  and  /6  correspond  to  a  system  of  4  equations  whose  unknowns  are  the  desired 
frequencies  of  location  in  the  X  —  Z  plane: 

f(XZ  =  00)  +  f(XZ  =  01)  =  3000 
f(XZ  =  10)  +  f(XZ  =  11)  =  7000 
f(XZ  =  00)  +  f(XZ  =  10)  =  5500 

f(XZ  =  01)  +  f(XZ  =  11)  =  4500 

The  maximum  entropy  solution  to  this  system  of  equations  is: 

X  Z  /7(.) 

0  0  1650 

0  1  1350 

1  0  3850 

1  1  3150 

The  entropy  of  (the  corresponding  relative  frequency  distribution  for)  f1  is  slightly  higher  than 
that  of  /4;  thus,  relative  to  distribution  /7,  the  amount  of  uncertainty  regarding  where  one  can 
expect  the  target  to  be  is  greater.  In  addition  to  their  difference  in  entropy,  /7  and  /4  are  seri¬ 
ously  in  conflict  qualitatively:  f\  gives  much  greater  frequency  to  XZ  =  00  than  to  XZ  =  01;  /7 
gives  greater  frequency  to  XZ  =  10  than  to  XZ  =  11. 

The  distributions  /4,  /42,  f\  and  /7  are  just  4  of  astronomically  many  frequency  distribu¬ 
tions  over  XZ  compatible  with  the  available  data.  It  will  be  argued  later  that,  in  a  decision-mak¬ 
ing  context,  it  is  not  necessary  to  pick  a  single  distribution.  However,  the  choice  of  sets  of  dis¬ 
tributions  will  make  a  difference.  Before  introducing  some  formal  notation  and  algebra,  the 
(extremely  rare)  case  in  which  one  can  deduce  the  frequencies  over  the  attributes  of  interest  will 
be  illustrated. 

Suppose  now  that  the  hypothetical  mountain  occupies  most  of  the  octants  XYZ  =  001, 
XYZ  =  101,  XYZ  =  000,  XYZ  =  Oil.  This  would  be  consistent  with  the  following  frequency 
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distribution: 


X  Y  Z  /„(.) 

0  0  0  0 

0  0  1  0 

0  10  0 

0  110 
10  0  0 

1  0  1  20 

1  1  0  50 

1  1  1  30 

As  is  the  case  with  any  joint  frequency  distribution  over  the  set  of  attributes  {X,Y,  Z),  there  is 
a  unique  set  of  marginal  distributions  compatible  with  it  (in  the  strict  sense  that,  e.g., 
f(XY  =  01)  =  f{XYZ  =  010)  +  f{XYZ  =  01 1)).  The  converse  is  almost  never  the  case  (as  the 
previous  example  illustrates).  The  marginal  distributions  (observed  data)  generated  by  f&  are: 


X 

Y 

/90) 

Y 

z 

/10C 

0 

0 

0 

0 

0 

0 

0 

1 

0 

0 

1 

20 

1 

0 

20 

1 

0 

50 

1 

1 

80 

1 

1 

30 

From  these  marginal  frequencies  it  can  be  deduced  that  the  joint  frequency  distribution  is  /8 
and  that  the  marginal  frequencies  of  interest  (over  { X,  Z })  are: 

X  Z  /„(,) 

0  0  0 

0  1  0 

1  0  50 

1  1  50 

Distribution  f&  is  identifiable  from  its  projections  (observed  data)  onto  {A,  7}  and 
{Y,  Z}.  It  is  rare  that  a  distribution  is  identifiable  from  projections.  If  the  frequencies  are 
required  to  sum  to  100,  then  there  are  274,436  distributions  over  {X,Y,  Z}  that  are  recon- 
structable  from  their  projections  onto  {A,  T}  and  {T,  Z).  However,  this  is  only  0.001%  of  the 
2. 6  x  1010  distributions  over  {X,  Y,  Z)  that  sum  to  100. 

In  what  follows,  methods  are  presented  for  dealing  with  the  uncertainty  that  is  virtually 
guaranteed  to  arise  concerning  the  frequencies  over  the  attributes  of  interest,  when  they  cannot 
be  observed  directly. 

B.  Definitions  and  algebra 

A  finite  product  space  is  the  Cartesian  product  of  a  finite  sequence  of  finite  sets.  In  what 
follows,  the  finite  sets  will  be  the  domains  of  database  attributes.  For  example,  the  discretized 
X  dimension  in  the  moving  target  example  is  a  attribute  with  domain  {0,  1}.  As  another 
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example,  the  domain  of  the  attribute  Number_of_doors  might  be  {1,2,  3, 4,  5}.  Although 
domains  needn’t  be  numeric,  e.g.,  {Male,  Female},  they  may  be  encoded  using  integers.  If  A  is 
an  attribute,  its  domain  will  be  denoted  dom(A).  The  finite  product  space  associated  with  a  set 
of  attributes  V,  where  IVI  =  n,  is 

n 

Sv  =  X  dom(Aj), 
j= i 

where 

<Aj,  A2)  9  A n ^ 

is  an  arbitrary  (but  fixed)  ordering  of  the  elements  of  V.  The  elements  t  of  Sv  are  n-tuples: 

t  =  <  d\,  a2, . . . ,  an>. 

Example: 

V  =  {Gender,  Dog_Owner} 
dom{  Gender)  =  {Male,  Female} 
dom (Dog_0 wner)  =  {Yes,  No} 

Sv  =  {<Male,  Yes>,  <Male,  No>,  <Female,  Yes>,  <Female,  No>}. 

If  WcY,  the  restriction  of  teSv  to  W,  denoted  t[VF],  is  the  unique  element  of  Sw  that 
agrees  with  t  on  all  elements  of  W.  For  example, 

<Male,  Yes>[{ Gender}]  =  <Male>, 
and,  for  <0,0,1>^S{X,y,z}  in  the  moving  target  example, 

<0,0,1>[{X,  Z}]  =  < 0,  1>. 

A  frequency  distribution  over  a  set  of  attributes  V  is  any  function  /  from  Sv  to  the  set  of 
nonnegative  integers: 

f:Sv  — »  N. 

A  distribution  will  usually  be  represented  in  the  form  of  a  table: 

Gender  Dog_Owner  /(.) 


Male 

Yes 

20 

Male 

No 

10 

Female 

Yes 

30 

Female 

No 

40 

In  the  statistics  literature,  such  tables  are  referred  to  as  contingency  tables  or  cross-tabulations 
[15].  As  in  the  moving  target  example,  the  symbol  /  may  be  sub-  or  superscripted.  The  set  V  is 

the  scheme  for  /,  denoted  s(f): 

s(f )  =  V. 

A  frequency  database  is  a  set  of  frequency  distributions,  for  example,  {/,,  f2}  from  the 
moving  target  example.  The  scheme  for  a  database  (its  database  scheme )  is  the  set  consisting  of 
the  schemes  of  its  elements.  If  D  is  a  database,  s(D)  denotes  its  scheme.  For  example, 

s({fuf2})={{X,Y},{Y,Z}}. 

Two  algebraic  operations  will  now  be  defined:  projection  and  extension. 

The  projection  of  /  onto  W,  where  WcF  =  s(f),  denoted  nw(f),  is  calculated  as 

M/)(0=  X  fit'). 

t’ zSv\t'[W]=t 
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The  projection  of  a  distribution  onto  a  subset  of  its  scheme  is  unique,  but  there  may  be  many 
other  distributions  sharing  that  projection.  In  the  moving  target  example, 

f\  =  nix,Yj(fl)  =  n{ x,Y)(fi )  =  ^/x.Yjifz) 


and 


fl  -  ^(Y.Zjifz)  ~  n{Y,Z)(fz)  ~  nIY,Z)(fl)- 

Lemma  1:  If  G  qH,  then  nG{nH{f))  =  nG(f). 


An  operation  complementary  to  projection  is  extension.  The  extension  of  /  to  Z,  where 
s{f)  c  Z,  denoted  Ez(f),  is  defined  as 

EZ(/)  =  {/V':  Sz  ->  N,  *«„(/')  =  /}. 


For  example, 

Lemma  2:  f  e  Es^\7tw(f)). 

Lemma  3:  nc(EH(f))  =  EG(f),  if  s(f)  cGcff. 


For  any  function  g:  A  — >  B,  the  image  of  C  cA  under  g,  g(C),  is  the  set  of  all  images  of 
elements  of  C : 

g(C)  =  { bib  =  g(x),  xeC}. 

So,  for  a  set  of  frequency  distributions  F,  where  W  c  s(f )  for  all  /  e  F, 

nw(E)=  {nw{f)\f  e  F}. 

Lemmas  4  and  5,  in  the  statement  of  which  SF  is  a  set  of  sets  of  frequency  distributions,  follow 
from  general  results  in  the  theory  of  functions  [16]: 


Lemma  4:  nw{  u  F)  =  u  nw{F). 

F  e  SF  F  e  SF 

Lemma  5:  nw{  n  F)  c  n  nw(F). 

F  eSF  F  e  SF 

The  function  symbols  'n'  and  'E'  may  be  overloaded,  e.g.,  to  apply  to  databases  in  addi¬ 
tion  to  individual  distributions  or  sets  of  distributions  sharing  the  same  scheme.  Some  prelimi¬ 
nary  definitions  are  necessary. 

Database  scheme  Sj  is  a  refinement  of  scheme  S2,  denoted  <  S2,  if  every  element  of  5) 
is  a  subset  of  some  element  of  S2. 


Example: 

{{*„  x2),  {X2,  X3},  {X3,  X4}}  <  {{X„  X2,  X4},  {X2,  X3,  X4}}. 

A  distribution  /  may  be  projected  onto  a  database  scheme  S  if  V  c  s(f)  for  all  V  e  S,  i.e.,  if 
S<{V): 

ns(f)={xv(fW  e  S}. 


Example: 

n dX,Y),(Y,Z})(fz )  =  (/i>  h)- 

If  Es(f\ns{f ))  =  {/},  then  /  is  identifiable  from  S. 

An  entire  database  may  be  projected  onto  a  refinement  of  its  database  scheme:  If  S  <  s(D), 

then 

*s(D)  =  {xv(f)\feD,  VeS,  V  c  s(f)}. 

Lemma  6  is  a  consequence  of  Lemma  1 . 
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Lemma  6:  If  Si  <  S2  ^  {.?(/)},  then 


XsMsiif))  =  *s,(/)- 

So,  in  the  moving  target  example, 

{/s>  /is}  =  nKX)jz})({ f\i  fii) 

=  X{(XUZ)}(X((X,Y)JY,Z  M 31)) 

=  X{(XUZI)(fl) 

Similarly,  an  entire  database  may  be  extended  to  a  set  of  attributes  that  is  a  superset  of  the 
union  of  the  elements  of  the  database  scheme: 

EZ(D)=  n  Ez(p), 

peD 

where  u  s(p )  c  Z. 

peD 

In  the  target  example,  E<x'Y'z,{{fx,  /2})  is  the  set  of  solutions  /3  to  the  system  of  8  integer 
linear  equations 

/3(<  0, 0, 0  >)  +  /3(<  0, 0, 1  >)  =  1500 
/3(<  0,l,0>)  +  /3«  0,1,1  >)  =  2000 


Theorem  9:  For  database  schemes  ^  and  S2,  Sx  <  S2  implies 

Ez^Sl(f))QEz(7TSl(f)). 

Proof:  The  elements  of  an  extension  are  solutions  to  a  system  of  linear  equations.  Each  of  the 
equations  determining  Ez(nS](f  ))  is  a  linear  combination  of  equations  from  the  system  deter¬ 
mining  Ez(xS2(f)).  Therefore,  every  solution  to  the  latter  system  is  also  a  solution  to  the  for¬ 
mer. 

Returning  to  the  moving  target  example,  recall  that  we  are  interested  in  frequencies  over 
the  scheme  {X,Z}.  We  have  available  the  database  {f\,  f2),  with  database  scheme 
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{{X,  Y],  {Y,  Zjj.  The  database  is  consistent  in  the  sense  that 

is  non-empty.  Thus,  it  is  reasonable  to  assume  that  there  is  a  distribution 

h*  6  E(XY'Z)([fu  f2}) 

such  that 

{fi’fl}  =  7r({X.YUY.Z))(h  *)• 

(This  is  analogous  to  the  universal  relation  assumption  in  relational  database  theory  [17].)  It 
happens  that  Efx,Y,zl([fl,  f2})  is  very  far  from  empty:  it  is  astronomically  huge.  Thus,  /3  *  is 
not  identifiable  from  its  projections  f\  and  f2.  If  it  were,  then  we  could  calculate  the  desired 
frequencies  as  n/Xi zjifi  *)• 

Assume  that  all  of  the  information  available  regarding  /3  *  is  embodied  in  the  database 
{ /i ,  /2 } .  In  Section  m.A,  the  construction  of  two  different  sets  of  distributions  (as  solution  sets 
to  systems  of  integer  linear  equations),  both  of  which  contain  the  unknown  distributions  /3  * 
was  illustrated.  One  of  the  sets  is  algebraically  represented  as 

*r{x,z)(E(XY'Z}({fi’  fi))- 

This  is  the  set  containing  /4,  /4,  and  /43,  among  many  others.  The  other  set  constructed  is 
It  contains  /7  (and  many  other  distributions). 

Although  /3  *  is  not  identifiable,  we  can  still  make  the  most  of  the  available  data  by  deriv¬ 
ing  from  it  as  much  information  as  possible  regarding  the  identity  of  the  distribution.  (Later  we 
will  discuss  tradeoffs  between  quality  of  information  and  cost  of  information  processing.) 
Which  of  the  two  approaches  discussed  so  far  retains  more  information  about  /3  *? 

Theorem  10:  Let  V  =  u  G.  Suppose  that  Y  is  a  cover  of  Vt,  V0  c  V.,  and  Y  <  s(D).  Then 

Ges(D) 

nVo{Ev{D))^KVo{Ev'{nY{D))). 

Proof:  If  EV(D)  =  0,  then 

xVq(Ev(D))  =  0c  kVq{Ev'{t:y{D))). 

Otherwise,  there  exists  an  /  such  that  D  -  7rs(D)(f).  Then 

nVo{Ev{D))  =  *Vo(£VW/))) 

=  ^v0(^v,(£;V(^(D)(/))))  [ Lemma  1] 

£  nVa{jtv^{Ev {nY (/))))  [Theorem  9] 

e  7rVo(Ev'(7tY(f)))  [ Theorem  8] 

=  nVo(EVl(7Ty(7rs{D)(f))))  [Lemma  6] 

=  xVi)(Ev'(7rY(D))) 

Substituting  {/„  f2)  for  D,  [X,  Y,  Z}  for  V,  {{X},  {Z}}  for  Y,  and  {X,  Z]  for  both  V0 
and  V]  in  Theorem  10,  we  obtain 
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*{xz0x**>({f\,  fi })  £  JC(x,z}(EIX'Z> (X({X},(Z}}(. { /i >  A)))) 

_  Elx'z>{n((xit(z)}{{f\,  h))), 

by  Lemma  3. 

From  an  information  content  standpoint,  the  set  of  distributions  resulting  from  the 
sequence  of  operations  extend,  then  project  (call  it  EP)  is  superior  to  the  set  (PE)  resulting  from 
the  sequence  project,  then  extend;  EP  is  guaranteed  to  be  a  subset  of  PE.  For  example,  although 
we  cannot  identify  ^x2y(/3*),  we  can  infer  bounds  on  its  components.  The  bounds  inferable 
from  EP  are  tighter  than  the  bounds  inferable  from  PE: 


X 

z 

min/e£P/(.) 

max/e£P  /(•  ) 

min/eP£/(.) 

ma  xfePE 

0 

0 

1500 

3000 

0 

3000 

0 

1 

0 

1500 

0 

3000 

1 

0 

2500 

4000 

2500 

5500 

1 

1 

3000 

4500 

1500 

4500 

However,  to  determine  the  tighter  bounds,  it  is  necessary  to  calculate  maxima  and  minima  rela¬ 
tive  to  a  system  of  8  equations  in  8  unknowns.  For  the  looser  bounds,  a  system  of  4  equations  in 
4  unknowns  is  sufficient.  Whether  the  additional  effort  is  worthwhile  can  only  be  determined  in 
the  context  of  the  use  to  which  the  information  is  to  be  put  (which  will  include  values  of  or  esti¬ 
mates  of  other  quantities,  e.g.,  utilities),  which  will  be  discussed  next. 

C.  Decisions  with  frequency  tables 

Recall  that  the  motivating  scenario  for  this  research  is  one  in  which  we  wish  to  hit  a  target 
by  dropping  an  object  on  it  from  above  (X  -  Z  plane).  We  have  recorded  frequencies  of  occu¬ 
pation  of  quadrants  in  the  X  -  Y  and  Y  -  Z  planes.  Without  additional  information  or  additional 
capabilities  (heat  seeking,  etc.),  the  probability  that  we  hit  the  target  will  be  highest  if  the  object 
is  dropped  into  the  X-Z  quadrant  whose  frequency  of  occupation  is  highest.  However,  we 
may  miss  the  target.  The  costs  associated  with  missing  may  vary  greatly  from  one  quadrant  to 
another.  These  costs  must  be  taken  into  account  also. 

The  decision  theory  developed  during  and  shortly  after  World  War  II  by  Von  Neumann  and 
Morgenstem  [18],  Savage  [19]  and  others,  is  the  dominant  approach  among  researchers  in  arti¬ 
ficial  intelligence  [20]  and  data  fusion  [21].  On  this  approach,  when  a  choice  is  to  be  made 
among  multiple  actions,  the  action  with  maximum  expected  utility  is  selected. 

On  this  model,  the  components  of  a  decision  problem  are  as  follows: 

•  A  set  of  mutually  exclusive  and  exhaustive  conditions  (states) 

C  —  { C\ ,  C2, . . . ,  cn} 

•  A  probability  function  on  C, 

p:C->[  0,1]. 

The  set  of  all  probability  functions  on  C  is  denoted  Pc. 

•  A  set  of  actions 

A  {  O,  j ,  #2  >  •  ■  •  >  tim } 
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A  utility  function 


u:AXC  — ^  R. 


The  elements  of  A  X  C  are  outcomes:  <a},  c f>  is  the  result  of  performing  action  a,  in  state  cr 
We  may  abbreviate  the  pair  <ah  Cj>  as  otj.  The  higher  the  value  of  «(0y),  the  more  desirable  is 
outcome  oi;. 

Example:  In  the  target  example, 

A  -  Moo>  ^oi>  ^10’  d\\ )>  where  dxz  stands  for  "drop  object  in  quadrant  xz  of  the  X  -  Z  plane"; 

C  =  [Iqq,  lQ i,  /10,  l\\ },  where  lxz  stands  for  "target  is  located  in  quadrant  xz  of  the  X  -  Z  plane"; 
and  u  is  given  by  the  table 


^00 

^01 

ho 

^ii 

roo 

0.8 

0 

0 

0 

roi 

0.3 

1.0 

0.3 

0.3 

f10 

0.5 

0.5 

0.9 

0.5 

fll 

0.6 

0.6 

0.6 

1.0 

(What  to  use  for  p  is  essentially  the  crux  of  this  report  and  is  discussed  below.) 

The  axioms  of  the  Von  Neumann  -  Morgenstem  theory  permit  assessing  utility  values 
("utiles")  on  a  scale  from  0  to  1,  assigning  1  to  the  most  preferred  outcome,  0  to  the  least  pre¬ 
ferred,  and  using  introspection  about  hypothetical  lotteries  to  assign  values  for  the  other  out¬ 
comes.  Outcomes  <d0l,  /01>  and  <du,  lu>  are  tied  for  most  preferred  because  region  l0]  is  a 
mountain,  /u  is  water,  lm  is  a  city,  and  /)0  is  sparsely  populated.  Similarly,  since  /00  is  a  city, 
outcomes  <d00,  /01>,  <d0 0,  /10>,  and  <dm,  ln>  are  tied  for  worst.  To  assess  the  utility  of 
<d0],  /10>,  for  example,  a  domain  expert  is  questioned  by  a  decision  analyst  (the  process  can  be 
and  has  been  automated)  regarding  preferences  between  <d0 ,,  /10>  for  certain  and  a  lottery  with 
two  "prizes":  <d0i,  /01>  (one  of  the  most  preferred  outcomes)  and  <dw,  ln>  (one  of  the  least 
preferred).  Initially,  the  probabilities  of  "winning"  each  prize  are  equal:  0.5.  If  a  preference  for 
<d0],  /10>  is  stated,  then  the  lottery  is  changed  to  give  probability  0.75  to  <d0l,  lQl>  (making 
the  new  lottery  more  desirable  than  the  old  one).  If  now  the  lottery  is  preferred,  the  probability 
of  <d0l,  Z01>  is  reduced  to  0.625.  (The  probabilities  are  usually  represented  graphically,  as  pie 
charts  or  "wheels  of  fortune".)  This  continues  until  the  domain  expert  expresses  indifference.  If 
p  is  the  final  probability  for  <<i0i.  lo\>’  then  it  follows  from  the  utility  axioms  that 

w(<  d0l,  /jo  >)  =  p  x  u(<  d0l,  l0 ,  >)  +  (1  -  p)  x  u(<  dm,  lu  >) 

=  p  x  1.0  +  (1  —  p)  x  0 
=  P- 

The  expected  utility  of  an  action  a,  is  computed  as 

n 

EU(a ,)  =  X  P(cj\ai)  x  M(<  ai,  Cj  >)• 

;=i 

Any  decision  problem  for  which  the  performance  of  an  action  affects  the  probability  of  a  condi¬ 
tion,  i.e.,  for  which 
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P(cj\at)  *  p{cj) 

can  be  transformed  into  an  equivalent  problem  in  which  the  actions  do  not  affect  the  probabili¬ 
ties  of  conditions  [22].  In  what  follows,  we  will  assume  that  our  decision  problems  have  the 


property 


i.e.,  that 


p(Cj\ai)  =  p(cj). 


EU (dj)  =  X  P(cj )  x  «(<  ah  Cj  >)). 
j= i 

For  our  moving  target  example,  we  need  probabilities  for  the  conditions  lxz.  Let  us  sup¬ 
pose,  again,  that  the  available  data  are  the  frequency  distributions  fx  and  f2,  each  of  which  is 
the  result  of  10,000  (coordinated)  observations: 


X  Y  m  Y  Z  /2(.) 

0  0  2000  0  0  4500 

0  1  1000  0  1  500 

1  0  3000  1  0  1000 

1  1  4000  1  1  4000 


So  far,  two  methods  of  obtaining  frequencies  over  the  attributes  {X,  Z}  of  interest  have  been 
discussed:  Pick  an  element  of 

X{x,z>(E{X'Y'ZI(ifi,  /2})), 

or  pick  an  element  of 

EfX'Z\x{{X),(Z)Mfi’  fz)))- 

We  have  identified  3  elements  (out  of  astronomically  many  altogether)  of  the  former  set  of  dis¬ 
tributions,  fl,  fl,  and  /43;  and  one  element  (out  of  astronomically  many)  of  the  latter,  /7.  One 
way  to  obtain  probabilities  from  these  frequencies  is  to  convert  them  to  relative  frequencies. 
Let  pj  denote  the  relative  frequency  distribution  associated  with  f( : 

pi(t°)=flj(to)/zlfij(t). 

Since,  for  this  example,  the  total  number  of  observations  (over  a  sample  space  of  four  elemen¬ 
tary  events)  is  10,000,  we  could  be  quite  confident  in  any  one  of  the  pj  ,  regarded  in  isolation, 
as  an  estimate  of  the  unknown  probability  of  occupation  of  quadrants  in  the  X  -  Z  plane.  95% 
confidence  intervals  for  p\,  for  example,  are  very  tight  [23]: 


X 

z 

/.'(•) 

Pit) 

95%  conf.  limits 

0 

0 

2500 

0.25 

+/-0.004 

0 

1 

500 

0.05 

+/-0.002 

1 

0 

3000 

0.30 

+/-0.005 

1 

1 

4000 

0.40 

+/-0.005 

Definition:  An  interval-valued  probability  distribution  over  a  set  of  attributes  V  is  a  mapping 

i:  Sy  — >  /, 

where  7  is  the  set  of  closed,  convex  subsets  (subintervals)  of  [0,1],  X  max  i(t)  >  1, 

tGzSy 

0<  X  min  i(t)  <  1.  We  may  then  associate  the  interval  distribution  i\  with  the  95% 

tESy 
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confidence  limits  on  the  values  of  p\: 

X  Z  »•*(.) 

0  0  [0.246, 0.254] 

0  1  [0.048, 0.052] 

1  0  [0.295, 0.305] 

1  1  [0.395, 0.405] 

We  may  do  the  same  for  distributions  /4,  /43  and  /7: 

X  Z  1 1  i\ 

0  0  [0.146,0.154]  [0.196,0.204] 

0  1  [0.146,0.154]  [0.097,0.103] 

1  0  [0.395,0.405]  [0.345,0.355] 

1  1  [0.295, 0.305]  [0.345, 0.355] 


_ h _ 

[0.161,0.169] 
[0.132,0.138] 
[0.380,  0.390] 
[0.310,  0.320] 


These  probability  estimates  are  in  serious  conflict:  not  only  are  the  4-way  intersections 

i\(t)  n  14(f)  n  il(t)  n  i7(f) 

empty  for  all  t,  so  are  each  of  the  6  2-way  intersections  -  no  interval  distribution  agrees  with 
any  other  on  any  of  the  4  values  t  e  S/X,z)- 

Definition:  i\:  Sv  — >  /  and  i2‘  Sv  — »  /  agree  onteSv  if 

i\(t)  n  i2(t)  *  0. 

/j  and  i2  agree  if  they  agree  on  all  t  e  Sv. 

The  expected  utilities  of  the  actions  d0 0,  d0l,  d]0,  and  dn,  relative  to  each  of  the  probabil¬ 
ity  distributions  p\,  p\,  p\,  and  p7  are  given  in  the  table  below: 


Pa 

2 

Pa 

pi 

Pi 

1 

00 

0.200 

0.120 

0.160 

0.132 

r 

01 

0.335 

0.206 

0.171 

0.395 

r 

10 

0.620 

0.660 

0.640 

0.654 

r 

11 

0.760 

0.412 

0.740 

0.726 

There  is  less  conflict  among  the  distributions  with  respect  to  the  action  identified  as  opti¬ 
mal.  Although  the  expected  utilities  for  a  given  action  relative  to  the  four  distributions  consid¬ 
ered  vary  widely,  three  of  the  four  give  maximum  expected  utility  to  action  dM.  This  leads  to 
the  concept  of  agreement  among  distributions  in  the  context  of  a  particular  decision-making 
scenario. 

The  domain  of  an  action  ah  denoted  D(a,),  is  the  set  of  probability  distributions  over  C 
relative  to  which  the  expected  utility  of  at  is  maximum  (or  tied  for  maximum)  among  all 
actions  under  consideration: 

n  n 

D(at )  =  {pePc I  X  P(cj)  x  u(<  ai -  cj  >)  ^  £  P(cj)  x  «(<  0,  Cj  >),  for  all  aeA). 
j= 1  J= 1 

To  illustrate,  the  domain  of  action  du,  D(du)y  is  the  set  of  all  solutions  p  to  the  system  of  lin¬ 
ear  equations: 
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p(loo )  +  PQ 01)  +  Piho)  +  PC^n)  -  1 


0. 6/?(/oo)  +  0-  6p(/0i)  +  0. 6p(Z10)  +  p(l\\)  ^  0. 8p(Zoo) 

0. 6p(loo )  +  0-  6p(/0i)  +  0. 6p(/10)  +  p(l\\)  ^  0. 3/?(/oo)  +  p(^oi)  +  0-  3p(ho) +  0-  3p(/n) 

0. 6 p(Iqo)  +  0. 6p(/01)  +  0. 6p(/10)  +  pihi)  —  0. 5p(/oo)  +  0. 5p(/oi)  +  0-  9p(fi o)  "h  5p(ln) 

Distributions  P4,  and  p7  are  elements  of  D(<Zn);  P4  is  not. 

Any  decision  problem  (with  components  C,  A,  and  u)  decomposes  Pc  into  convex  subsets 
Dia^),  D(a2),  D(am): 

Pc  =  u£)(flf). 

1=1 

The  decomposition  is  not  a  partition,  however,  since  the  domains  (convex  polyhedra)  can  share 
faces. 

In  the  context  of  a  decision  problem,  distributions  p  and  p'  are  equivalent  if  they  belong  to 
the  same  domain(s).  Thus,  modulo  this  problem,  despite  diverging  dramatically  (e.g.,  p}(/0i)  = 
0.050,  Pnikx)  =  0.135),  p\,  p\  and  p7  are  equivalent.  Note,  though,  that  p\,  p\,  p\  and  p7  are 
just  4  of  infinitely  many  distributions  over  {X,  Z]  that  are  compatible  in  some  sense  with  the 
original  frequency  data.  That  3  of  these  4  belong  to  D(du)  is  not  sufficient  reason  to  perform 
action  dn. 


D.  Data  Consistency 


It  is  assumed  in  the  moving  target  example  that  the  observations  are  coordinated,  accurate, 
and  correctly  recorded.  As  a  result,  the  database  is  consistent.  A  database  D  is  globally  consis¬ 
tent  if 

EV(D)  *  0, 

where  V  =  u  W.  A  database  D  =  {/„  f2,  •■•/*}  is  locally  consistent  if 

Wes(D) 

-  ns(f,)r,s(fj)(f  jb 

for  all  fh  fj  e  D.  (The  distributions  are  said  to  be  pairwise  consistent.) 

If  s(D),  viewed  as  a  hypergraph,  is  or-acyclic  [10],  then  if  D  is  locally  consistent,  it  is  also 
globally  consistent.  It  is  computationally  cheaper  to  test  for  pairwise  (local)  consistency  of  the 
elements  of  D  than  to  check  for  a  feasible  solution  to  the  system  of  integer  linear  equations  cor¬ 


responding  to  E{D).  (The  latter  is  an  NP-complete  problem  [2].) 

To  test  for  a-acyclicity,  apply  step  (0)  to  s(D),  then  steps  (1)  and  (2),  in  any  order,  until 


neither  has  any  effect  on  G: 


(0)  G  =  s(D) 

(1)  If  v  is  contained  in  only  one  V  e  G,  V  =  V  -  {vt } 

(2)  If  V\  e  G  and  V2  e  G  and  Vj  qV2,G  =  G-  { Vx } 

If  the  final  value  of  G  is  {0},  then  s{D)  is  a-acyclic;  otherwise,  s(D)  is  «-cyclic  [10]. 
Example:  { {X,  Y},  {7,  Z) }  is  a-acyclic: 

(a) ({X,F},!F,Z}} 

(b) {{F},{y,zn 
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(c) {{T,Z}} 

(d)  { { Z } } 

(e)  {0} 

Example:  { { X,  Y } ,  { Y ,  Z } ,  { X,  Z } }  is  cr-cyclic: 

(a  ){{X,Y},{Y,Z},{X,Z}} 

Neither  step  (1)  nor  step  (2)  is  applicable. 

Since  { {X,  Y },  {Y,  Zjj  is  or-acyclic,  the  moving  target  database  can  be  tested  for  (global) 
consistency  by  testing  local  consistency: 

Y  *,y/(/i)(.) _ xmiM. ) 

0  2000  +  3000  =  5000  5000  =  4500  +  500 

1  1000  +  4000  =  5000  5000  =  1 000  +  4000 

The  algebra  presented  so  far  applies  also  to  the  relative  frequency  distributions  p  derivable 
from  frequency  distributions  /  as 

p(t)  =  /(O/Xf  fit)- 

The  same  basic  arithmetic  operation,  addition,  underlies  the  operations  of  projection  and  exten¬ 
sion  for  frequencies  and  probabilities.  An  additional  operation,  join,  involving  multiplication 
and  real-number  division,  is  meaningful  for  probability  distributions.  The  join  of  two  distribu¬ 
tions  px  and  p2  is  calculated  as 

•/(Pi,  Pi)(t)  =  Pi(a)x  p2(b)/  2  P2(c)> 

clc[j(pi)r\*(P2))  =  i>(i(pi)Ttf(p2)l 

where  a  =  t[s(/?i)],  b  -  f[s(p2)]>  and  where  the  denominator  equals  1  if  s(px)  n  s(p2)  =  0. 

Calculating  p(t)  in  this  way  amounts  to  the  assumption  of  conditional  independence  of 
s(P\)  ~  {s(P\)  n  s(p2 ))  and  s{p2)  -  (j(p,)  n  s(p2)),  given  s(p{)  n  s(p2)  [20],  J{px,  p2)  is 
also  the  maximum  entropy  element  of  ES<'P])KJS<'P2\{  pu  p2})  [24], 

Example:  Let  px  and  p2  be  the  relative  frequency  distributions  associated  with  fx  and  f2  from 
the  moving  target  example.  Then  J{px,  p2)  is  calculated  as  in  the  table  below: 


X  Y  Z _ JiPuPzM 

0  0  0  0.18  =  0.20x0.45/(0.45  +  0.05) 

0  0  1  0.02  =  0.20x0.05/(0.45  +  0.05) 

0  1  0  0.02  =  0.10x0.10/(0.10  +  0.40) 

0  1  1  0.08  =  0.10x0.40/(0.10  +  0.40) 

1  0  0  0.27  =  0.30x0.45/(0.45  +  0.05) 

1  0  1  0.03  =  0.30x0.05/(0.45  +  0.05) 

1  1  0  0.08  =  0.40x0.10/(0.10  +  0.40) 

1  1  1  0.32  =  0.40x0.40/(0.10  +  0.40) 


If  s(D),  where  D  =  {/),  f2,  •••,/*},  is  or-acyclic,  then  the  maximum  entropy  element  of 
E({p\,  p2,  ■■■  ,  pk})  can  be  calculated  by  a  sequence  of  applications  of  the  join  operation: 
J(Po(k)>  J(Po(k- 1)>  •  •  • ,  J (Pa(2)i  Pad))  •  *  0).  where  s{pa( ,>)  is  the  first  element  of  s(D)  elimi¬ 
nated  by  an  application  of  the  test  for  acyclicity,  s(pam)  is  the  second,  etc.  If  s(D)  is  a-cyclic, 
calculation  of  the  maximum  entropy  element  of  E{{px,  p2  ,  •  •  •  ,  pk})  requires  application  of 
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an  expensive  iterative  proportional  fitting  algorithm  [25], 

Thus,  there  are  computational  advantages  to  databases  with  a -acyclic  schemes.  On  the 
other  hand,  there  are  informational  advantages  to  cyclic  schemes. 

Theorem  11:  For  any  distribution  /  and  any  database  D  =  ns{D)(f  )  over  set  of  attributes  s(  f  ), 
where  s(D )  *  {s(f)}  and  \s(f)\  >  3,  if  s(D)  is  a-acyclic,  then  there  exists  a  database  D'  over 
s(f)  such  that 

(1)  s(D')  is  a -cyclic; 

(2)  D'  =  nsm{fy, 
and 

(3)  Es(f\D')  c  Es(f\D). 

Proof:  First,  note  that  if  s(D )  =  {s(f)},  then  D  =  {/}.  If  s(D)  *  {s(f)}  and  s(D )  is  a-acyclic, 
let 

D'  =  {K\K  c  s(f)  and  1/sTI  =  \s(f)\  -  1 } . 

D'  is  a-cyclic:  Each  attribute  is  contained  in  ls(/)l  -  1  elements  of  D'.  Since  \s(J)\  >  3, 
\s(f)\  -  1  >  2,  and  no  attribute  is  contained  in  only  one  element  of  D'\  so  step  (1)  of  the  cyclic¬ 
ity  test  algorithm  is  not  applicable.  Since  no  element  of  D'  is  a  subset  of  any  other  element  of 
D',  step  (2)  is  also  inapplicable.  D  is  a  refinement  of  D':  for  every  element  V  e  D  there  is  a 
V'  e  D'  such  that  V  c  V'.  Therefore,  by  Theorem  9, 

Es(f\nsm{f))  c  Esif\xs(D)(f)), 

which  implies 

Esif\D')  c  E5{f)(D). 


Example:  Suppose  the  actual  frequency  distribution  (over  100  time  units)  is 


X  Y  Z  /12(.) 

0  0  0  50 

0  0  1  0 

0  1  0  20 

0  11  0 

10  0  0 

110  5 

1  1  1  25 


If  the  available  data  are  and  n(Y,z}(fn\ the  inferable  bounds  on  fn  are 
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X  Y  Z  min  max 


0 

0 

0 

45 

50 

0 

0 

1 

0 

5 

0 

1 

0 

0 

20 

0 

1 

1 

0 

20 

1 

0 

0 

0 

5 

1 

1 

0 

5 

20 

1 

1 

1 

5 

25 

However,  fn  is  identifiable  from  its  projections  onto  {{X,Y},  {Y,  Z},  {X,  Z}}.  On  the  other 
hand,  although  observations  over  the  scheme  {{X,  Y},  {Y,  Z),  {X,  Z}}  provide  more  informa¬ 
tion,  obtaining  the  additional  data  may  be  too  expensive,  too  dangerous,  or  impossible.  Further¬ 
more,  a  system  of  12  equations  is  required  for  the  less  refined  scheme,  vs.  8  equations  for  the 
more  refined  scheme. 

It  is  quite  likely  that  frequencies  obtained  experimentally  vs.  by  projection  from  a  joint 
frequency  distribution  over  any  scheme  that  is  not  a  partition,  even  if  it  is  a-acyclic,  will  be 
incosistent. 

Example:  Suppose  that  coin  c2  is  tossed  250  times  in  conjunction  with  coin  c,  and  another  250 
times  in  conjunction  with  coin  c3,  and  the  joint  toss  outcomes  are  recorded  in  the  table  below. 


ci  c2  /13C) 

H  H  70 

H  T  60 

T  H  40 

T  T  80 


c2  c3  /14C) 

H  H  65 

H  T  50 

T  H  70 

T  T  65 


The  database  {/13,  /14]  is  inconsistent,  as  is  the  database  of  relative  frequencies 

ci  c2  P13O  c2  c3  pu(.) 


H 

H 

0.28 

H 

H 

0.26 

H 

T 

0.24 

H 

T 

0.20 

T 

H 

0.16 

T 

H 

0.28 

T 

T 

0.32 

T 

T 

0.26 

Distributions  p13  and  p14  give  conflicting  information  on  the  bias  of  coin  c2: 

XfciliPli)  *  X(c2}(Pu)- 

However,  standard  95%  confidence  intervals  for  p13  and  p14  are  consistent: 


Ci 

c2 

*13(-) 

c2 

c3 

jm(-) 

H 

H 

[0.224,  0.336] 

H 

H 

[0.206,  0.314] 

H 

T 

[0.187,0.293] 

H 

T 

[0.150,  0.250] 

T 

H 

[0.115,0.205] 

T 

H 

[0.224,  0.336] 

T 

T 

[0.262,  0.378] 

T 

T 

[0.206,  0.314] 

A  pair  of  interval  distributions  i  and  1  is  consistent  if 

*j(.W0(O  n  ^(o™(«')(i')(0  *  0. 
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for  all  t  over  s(i)  n  s(i').  (Note  that  this  is  a  more  general  concept  than  agreement,  as  defined  in 
Section  ffl.C.)  The  projection  of  an  interval  distribution  is  obtained  via  linear  programming. 
For  example,  the  endpoints  of 

n{C2)(i  n)(H) 

are  obtained  by  minimizing  and  maximizing  p(H)  subject  to  the  constraints 

p(H)  =  p(HH )  +  p(HT ) 

P(T)  =  p(TH)  +  p(TT) 
p(H)  +  p(T)=  1 
p{HH)>  0.224 
p(HH )  <  0. 336 


p(TT)  >  0. 262 
p{TT)  <  0. 378 

The  remaining  endpoints  are  obtained  similarly: 

<?2  ^0'l3)(-) _ ^;(^~14)(-  ) 

H  [0.339,0.541]  [0.356,0.564] 

T  [0.459, 0.661]  [0.436, 0.644] 

(As  this  example  illustrates,  the  endpoints  for  a  projection  are  not  always  sums  of  endpoints  of 
the  projected  intervals.)  Since  the  intersections  are  all  non-empty,  z13  and  iH  are  consistent. 

The  (real-valued)  extension  of  an  interval  database  /  =  {*'i,  ••  • ,  z*},  EV(I),  is  the  set  of  all 
(real-valued)  probability  distributions  over  V  whose  (real-valued)  projections  onto  s(I )  satisfy 
the  inequality  constraints  implied  by  the  elements  of  /. 

Example:  £,/rci,C2’C3;({/13,  z14})  is  the  set  of  all  solutions  p  to  the  system  of  inequalities: 

p(HHH)  +  p(HHT)  >  0. 224 

p(HHH)  +  p(HHT)  <  0. 336 
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p(HTT)  +  piTTT)  >  0. 206 
p(HTT)  +  p(TTT)  <  0. 314 
p(HHH)  +  •  •  •  +  p(TTT)  =  I 

An  interval  database  /  is  globally  consistent  if  EV(I )  *  0,  where  V  =  us(/).  If  5(7)  is 

i  €  / 

or-acyclic,  local  consistency  of  I  is  sufficient  for  global  consistency. 

V.  Decisions  with  interval  databases. 

Frequency  and  relative  frequency  databases  are  likely  to  be  inconsistent.  We  may  replace 
them  with  interval  databases,  which  are  much  less  likely  to  be  inconsistent.  We  may  then  work 
with  probability  intervals  or  sets  of  (real-valued)  probability  distributions  in  selecting  an  opti¬ 
mal  action.  This  requires  adaptation  of  the  classical  maximum  expected  utility  criterion. 

In  the  moving  target  example,  suppose  we  have  far  fewer  than  10,000  observations  and 
that  the  database  is  inconsistent  (due  perhaps  to  differences  in  perspective,  when  the  object  is 


not  entirely  within  a  single  X  - 

X 

Y-Z  cell): 

Y  /15(.) 

Y 

Z 

/>6(.) 

o” 

0 

10 

0 

0 

25 

0 

1 

25 

0 

1 

5 

1 

0 

15 

1 

0 

60 

1 

95%  intervals: 

1 

50 

1 

1 

10 

X 

Y 

‘•?f  (•) 

Y 

Z 

495(.) 

0 

0 

[0.041,0.159] 

0 

0 

[0.165,0.335] 

0 

1 

[0.165,0.335] 

0 

1 

[0.007,  0.093] 

1 

0 

[0.106, 0.194] 

1 

0 

[0.539,  0.661] 

1 

1 

[0.402,  0.598] 

1 

1 

[0.041,0.159] 

We  may  project  E(X,Y'z,({i0^5,  i°695  } )  onto  {X,  Z)  via  linear  programming  to  construct  the 
interval  distribution: 

X  Z  i,7(.) 

0  0  [0.006, 0.492] 

0  1  [0.000, 0.252] 

1  0  [0.256, 0.792] 

1  1  [0.000, 0.252] 

With  the  set  of  actions,  conditions  and  utility  function  of  Section  III.C,  these  probability 

intervals  determine  expected  utility  intervals: 
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EU(doo)  e  [0. 0048,  0. 3936] 

EU(d0l)  6  [0.3000,  0.4764] 

EU(dl0)  e  [0.6024,  0.8168] 

EU(dn)  e  [0.6000,  0.7008] 

It  is  now  possible  to  eliminate  actions  dm  and  d0l  from  consideration:  they  are  both  dominated, 
in  the  sense  that  there  exist  actions  (dl0  and  dn)  whose  minimum  expected  utility  is  greater 
than  their  maximum  expected  utility. 

We  need  to  choose  between  dl0  and  dn.  If  additional  sampling  is  possible,  it  will  narrow 
the  probability  intervals,  which  in  turn  will  narrow  the  utility  intervals,  increasing  the  likelihood 
that  a  single  action  will  be  undominated. 

If  additional  sampling  is  not  possible,  we  may  narrow  the  intervals  by  choosing  a  lower 
confidence  level.  This  gives  a  "context-driven"  solution  to  how  conservatively  we  should  treat 
our  observed  relative  frequencies  as  estimates  of  probabilities.  A  second  alternative  is  to  base 
the  choice  between  the  undominated  actions  on  their  security  levels  [26]:  choose  the  (undomi¬ 
nated)  action  whose  minimum  utility  is  maximum  ("maximin").  This  policy  would  select  du 
(as  it  would  if  we  had  no  frequency  data  whatsoever). 

The  algebraic  results  in  Section  III.B  permit  additional  strategies  that  are  useful  when 
dealing  with  larger  numbers  of  attributes.  In  a  manner  analogous  to  iterative  deepening  search, 
we  may  project  the  database  onto  a  sequence  of  increasingly  less  refined  schemes,  starting  with 

{  {  v}lv  6  V], 

where  V  is  the  set  of  attributes  directly  relevant  to  the  decision  problem.  It  follows  from  Theo¬ 
rem  10  that  the  utility  intervals  will  become  narrower  at  each  step  in  the  sequence.  However, 
the  number  of  variables  and  inequalities  in  the  corresponding  linear  program  increases  also 
with  each  iteration. 
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FIGURES 

Intersecting  (case  ii)  vs.  Nonintersecting  (case  i)  Marginal  Constraints 
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Backtracks  -  25th  Percentile 


Backtracks  -  25th  Percentile 


Cyclic  Database  Schemes 


Cyclic  scheme,  5  attributes,  7  marginals 
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Database  Inconsistency  Problem 


Figure  12 
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Abstract 

Automated  target  recognition  has  benefited  from  cross-fertilization  of  development  in  related  subdisciplines  of 
image  processing  such  as  medical  imaging.  For  example,  the  application  of  computerized  tomography  to  synthetic 
aperture  radar  (SAR)  imaging  has  produced  3-D  reconstructions  of  ground  targets  on  an  experimental  basis.  In  prac¬ 
tice,  by  acquiring  multiple  views  of  a  target  (also  called  multi-look  imaging  -  MLI)  that  are  subsequently  merged  math¬ 
ematically,  one  can  obtain  reasonable  approximations  to  higher-dimensional  reconstructions  of  a  target  of  interest.  For 
example,  multiple  two-dimensional  airborne  images  of  ground  objects  can  be  merged  via  the  Fourier  transform  (FT) 
to  obtain  one  or  more  approximate  three-dimensional  object  reconstructions.  Additional  methods  of  3D  model  con¬ 
struction  (e.g.,  from  affine  structure)  present  advantages  of  computational  efficiency,  but  are  sensitive  to  positioning 
errors. 

In  this  study,  an  analysis  of  MLI  is  presented  that  applies  to  various  scenarios  of  nadir,  near-nadir,  or  off-nadir 
viewing  with  a  small  or  large  number  of  narrow-  or  wide-angle  views.  A  model  of  imaging  through  cover  describes 
the  visibility  of  a  given  target  under  various  viewing  conditions.  The  model  can  be  perturbed  to  obtain  theoretical  and 
simulated  predictions  of  target  reconstruction  error  due  to  (a)  geometric  projection  error,  (b)  focal-plane  quantization 
error  and  camera  noise,  (c)  possible  sensor  platform  errors,  and  (d)  coverage  of  looks.  An  information-theoretic  model 
is  derived  from  the  imaging  model  that  can  facilitate  prediction  of  limiting  sensor  geometry  and  view  redundancy  un¬ 
der  various  imaging  constraints  (e.g.,  target  and  cover  geometry,  available  range  of  look  angles,  etc.).  Mathematical 
models  are  implemented  in  a  software  simulation  model  that  is  coded  in  C  and  runs  on  UNIX  platforms.  Additional 
research  emphasizes  analysis  of  several  target  reconstruction  algorithms  based  on  estimated  errors  in  MLI  data  and 
systematic  error  sources  such  as  computational  error  in  arithmetic  operations.  Implementational  discussion  concerns 
the  employment  of  the  computer  simulation  model  developed  under  the  scope  of  this  study  in  applications  such  as  lo¬ 
cation  of  submerged  targets  from  airborne  imagery,  and  prediction  of  optimum  sensor  track  given  partial  knowledge 
of  target  position  and  size  as  well  as  cover  geometry  and  location. 

Study  notation  is  a  subset  of  image  algebra,  a  rigorous,  concise,  computationally  complete  notation  that  unifies 
linear  and  nonlinear  mathematics  in  the  image  domain  [Rit96],  Image  algebra  was  developed  at  University  of  Florida 
over  the  past  decade  under  the  sponsorship  of  DARPA  and  the  US  Air  Force,  and  has  been  implemented  on  numerous 
sequential  workstations  and  parallel  processors.  Hence,  our  algorithms  are  rigorous  and  widely  portable. 
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ERRORS  INHERENT  IN 
RECONSTRUCTION  OF  TARGETS  FROM 
MULTI-LOOK  IMAGERY 

Mark  S.  Schmalz 

1 .  Introduction 

Automated  recognition  of  ground  targets  from  airborne  imagery  can  be  confounded  by  vegetative  or  manmade 
cover  that  overlies  targets  of  interest.  For  example,  consider  a  vehicle  parked  beneath  an  overhanging  tree  line  or  con¬ 
cealed  under  camouflage  netting.  Under  such  circumstances,  it  would  be  advantageous  to  acquire  and  exploit  multiple 
target  images  at  different  look  angles  (called  multi-look  imaging  or  MLI)  [Lon81,Chr96,Moo96,Sha96,PAR97].  Se¬ 
lected  MLI  images  could  be  combined  mathematically  to  yield  a  composite  view  of  a  target  of  interest.  With  appro¬ 
priate  supporting  models,  this  process  is  being  implemented  in  computer  software  that  produces  simulated  MLI 
imagery  and  error  analyses  [Sch97a,b]. 

In  this  study,  the  following  objectives  are  accomplished: 

1)  Understand  and  model  the  phenomenology  of  multi-look  imaging; 

2)  Develop  models  and  error  analyses  for  tomographic  or  tomography-like  target  reconstruction  from  multiple 
airborne  views  of  ground  targets; 

3)  Apply  theory  and  models  to  the  development  of  computer  software  that  estimates  errors  in  salient  system  pa¬ 
rameters  under  given  sensing  geometry  constraints;  and 

4)  Acquire  realistic  laboratory  or  field  imagery  to  facilitate  algorithm  design  and  test. 

Computer  model(s)  are  summarized  in  Section  4. 

1.1.  Document  Organization 

This  report  begins  with  a  review  of  previous  work  (Section  1.2)  and  a  summary  of  key  study  issues  and  assump¬ 
tions  (Section  2),  followed  by  a  presentation  of  theory  and  methodology  that  describes  basic  MLI  phenomenology 
(Section  3).  Theory  is  extended  to  yield  an  information-theoretic  model  of  target  visibility  in  MLI  scenarios,  which 
supports  theory  development  for  tomographic  reconstruction.  The  result  of  this  study,  a  preliminary  analysis  of  the 
difficult  and  error-prone  process  of  target  reconstruction  from  multiple  views  is  discussed  in  Section  4.  Conclusions 
and  suggestions  for  future  work  are  presented  in  Section  5. 

1 .2.  Previous  Work 

Detection  and  recognition  of  manufactured  targets  in  natural  scenes  is  based  on  the  selection  of  a  model  that  best 
matches  a  subregion  of  an  observed  image.  When  an  imaging  sensor’s  view  of  a  target  is  unobstructed,  the  matching 
process  can  be  implemented  ia  1)  least-squares  minimization  in  image  space,  or  2)  suboptimal  methods  using  corre¬ 
spondence  subsets.  In  the  presence  of  partial  occlusion,  one  could  3)  progressively  reconstruct  the  object  via  coregis¬ 
tration  and  merging  of  multiple  views  or  4)  employ  tomography-like  methods  that  approximate  a  solid  model  of  the 
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target.  The  tomographic  model  could  be  serially  sectioned  with  respect  to  a  reference  vector,  thereby  yielding  a  lay¬ 
ered  model  of  objects  located  between  ground  and  cover.  Methods  1)  through  3)  arc  summarized  in  Sections  1.2.1- 
1.2.3,  followed  by  comparison  of  three-dimensional  (3-D)  models  and  2-D  images  in  Section  1.2.4.  We  do  not  con¬ 
sider  the  limiting  and  extremely  difficult  case  of  total  occlusion,  which  is  outside  the  scope  of  this  study. 

1.2.1.  Least-squares  minimization  techniques.  Traditional  photometric  approaches  to  matching  images  and  reference 
models  involve  least-squares  minimization  of  the  Euclidean  distance  between  the  model  and  an  elevation  map  con¬ 
structed  from  the  image.  This  problem  is  generally  classified  as  an  exterior  orientation  calibration  problem  (EOCP) 
whose  solution,  given  a  set  of  3-D  (2-D)  model  (image)  points  U  (V),  finds  a  rigid  transformation  that  minimizes  a 
distance  between  U  and  V  [Bas96].  Since  analytical  solutions  to  the  EOCP  remain  undiscovered,  numerical  methods 
are  employed,  which  tend  to  be  unstable  in  the  presence  of  discontinuities  and  noise,  are  computationally  costly,  and 
require  accurate  initial  values. 

Marr  and  Nishihara  [Mar78]  first  addressed  the  numerical  problems  of  describing  3-D  structure  systematically. 
Minsky  [Min75]  had  previously  suggested  that  the  complexity  of  shape  description  could  be  minimized  by  appropriate 
choice  of  primitives,  although  this  has  yet  to  be  realized  in  general  theory.  In  the  intervening  two  decades,  surface 
reconstruction  from  a  single  view  has  been  implemented  in  terms  of  variational  methods  [Mei79,Sch77,Ter83,Ter88], 
segmentation  and  synthesis  or  a  variety  of  ad  hoc  techniques  [Bes85].  Bolle  and  Vesuri  [Bol91]  summarize  varia¬ 
tional  surface  fitting  techniques  that  (a)  have  varying  degrees  of  viewpoint  invariance  as  well  as  robustness  in  the  pres¬ 
ence  of  noise,  (b)  tolerate  bias  in  parameter  estimates,  and  (c)  exhibit  various  sensitivities  to  obscuration.  The 
following  discussion  is  derived  from  the  excellent  overviews  given  in  [Bol91]  and  [Vem86,87]. 

In  its  simplest  form  [Bar81],  surface  fitting  involves  interpolation  of  uniformly  curved  surfaces  from  initial  ori¬ 
entation  values  and  constraints  using  a  relaxation  algorithm  based  on  parallel  iterative  local  averaging.  Unfortunately, 
this  technique  ignores  surface  discontinuities,  is  computationally  costly,  and  produces  a  surface  representation  that  is 
invariant  to  3-D  rigid  motion. 

Grimson  [Gri8 1  ]  presented  a  theory  of  visual  surface  interpolation  based  on  range  data  from  stereo  imagery  that 

were  obtained  via  Marr  and  Poggio’s  correspondence  algorithm  [Mar79].  Given  a  finite  image  domain  XcRn  and 

x 

an  elevation  map  a  e  R  s=  X  — »  R  ,  Grimson  minimized  the  variation  of  the  quadratic  functional 

E  =  Jaxx  +  2axy  +  ayy  dx  *3  (D 

N 

where  (x,  y )  €  X  ,  N  c  X ,  and  axy  =  dx/dy  .  E  denotes  the  energy  of  a  thin  plate,  hence  its  minimization  yields  a 

function  called  thin  plate  splines  [Mei79].  Grimson  developed  an  iterative  algorithm  based  on  biharmonic  conver¬ 
gence  that  results  from  applying  Euler’s  equations  to  minimize  E.  Although  invariant  to  rotation  and  translation,  Equa¬ 
tion  (I)  is  not  invariant  to  viewpoint,  since  E  depends  on  the  coordinate  system  in  which  depth  constraints  are  specified. 
Additionally,  Grimson’s  model  did  not  admit  discontinuities. 
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In  contrast,  Terzopoulos  [Ter83,88]  derived  an  efficient  surface  model  that  accepts  multiresolution  imagery,  is 
computationally  efficient,  and  accounts  for  depth  and  orientation  discontinuities.  Surfaces  are  modeled  as  thin  plate 
segments  joined  by  membrane  strips  along  orientation  discontinuities  and  bounded  by  depth  discontinuities.  The  mod¬ 
el  is  invariant  to  translation  and  rotation  but  not  to  rigid  motion,  and  is  based  upon  a  continuity  stabilizer  described  in 
[Ter83],  Harris’  extension  [Har86]  of  Terzopoulos’  model  couples  depth  and  slope,  this  integrating  orientation  con¬ 
straints  into  the  minimization  of  the  energy  functional  E.  Harris’  model  admits  varying  surface  smoothness,  parallel 
implementation,  discontinuities  of  any  order,  and  generalizes  splines  under  tension  since  it  can  integrate  arbitrary  com¬ 
binations  of  membrane,  thin-plate,  or  high-order  smoothness  constraints. 

In  MLI  imagery,  the  surface  fitting  method  would  be  complicated  by  the  lack  of  registration  between  views  that 
could  exhibit  geometric  distortion,  rotation,  and  scale  differences.  Hence,  a  parameterized  method  (e.g.,  quadric  or 
superquadric  representation)  would  be  required  that  could  be  scaled,  rotated,  and  translated  to  implement  coregistra¬ 
tion  between  views  in  model  space  (versus  customary  image-domain  coregistration).  Sensor  noise,  projection  error, 
and  quantization  error  would  thus  increase  surface  fitting  error.  More  importantly,  the  greyscale  variations  induced 
by  patterns  of  light  and  shadow  or  camouflage  devices  could  easily  confound  the  construction  of  an  elevation  map 
based  on  well-known  shape-from-shading  techniques,  as  discussed  in  [Sch97b], 

1.2,2.  Suboptimal  matching  with  correspondence  subsets.  The  change  in  the  2-D  projection  of  a  moving  3-D  object 
or  camera  yields  important  information  for  3-D  object  reconstruction  that  elucidates  relative  geometry.  The  problem 
of  computing  shape  from  motion  or  a  sequence  of  images  has  been  addressed  using  calibrated  or  uncalibrated  cameras 
[Fau92]  as  well  as  projective  or  affine  reconstruction  models  [Fau95],  Camera  calibration  facilitates  the  computation 
of  Euclidean  shape  up  to  a  scale  factor  using  projective  or  affine  models  [Der94],  In  the  absence  of  calibration,  the 
recovered  shape  is  defined  up  to  a  projective  or  affine  transformation.  Although  weaker  than  that  provided  by  surface 
fitting  from  depth  or  elevation  maps,  or  by  inversion  of  projection  equations  with  fully  specified  camera  calibration 
matrices,  relative  geometry  is  useful  since 

1 .  Reconstruction  algorithms  can  be  made  simpler  and  more  easily  parallelized. 

2.  The  costly  and  time-consuming  process  of  camera  calibration  is  not  required; 

3.  Multiple  views  of  an  object  need  not  be  equally  spaced  to  obtain  regularity;  and 

4.  Distinction  between  orthographic  and  perspective  projections  is  not  required. 

In  practice,  one  (a)  chooses  a  representation  of  projective  space  in  which  an  arbitrary  reference  plane  is  assumed  to  be 
located  at  infinity,  (b)  describes  the  new  representation  by  an  element  of  an  affine  group  applied  to  the  initial  repre¬ 
sentation,  and  thus  (c)  obtains  an  affine  invariant  relative  to  the  initial  representation. 

Affine  reconstruction  is  expected  to  be  a  useful  target  reconstruction  technique  in  conjunction  with  MLI.  Three 
recent  papers  highlight  associated  technical  issues: 
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•  Christy  &  Horaud  [Chr96]  incrementally  perform  Euclidean  reconstruction  based  on  a  weak-perspective  or 
paraperspective  (calibrated)  camera  model.  Fast  convergence  (in  a  few  iterations),  computational  efficiency, 
and  solution  of  the  sign  reversal  ambiguity  problem  are  cited  as  advantages 

•  Moons  et  al  [Moo96]  describe  affine  reconstruction  of  a  scene  from  two  views  with  relative  target-camera 
translation  between  views.  The  reported  approach  requires  only  five  corresponding  points,  which  can  be  ob¬ 
tained  by  well-known  coregistration  algorithms  [Yes89,Smi96],  unless  significant  anisomorphic  distortion  is 
present  in  coregistered  views.  This  is  usually  the  case  for  multi-look  imagery  where  a  majority  of  the  views 
are  taken  over  a  narrow  angular  range. 

•  Shashua  &  Navab  [Sha96]  describe  an  affine  reconstruction  technique  for  perspective  views  that  unifies  Eu¬ 
clidean,  projective,  and  affine  models.  Simple  algorithms  are  presented  for  reconstruction  from  multiple 
views,  recognition  by  alignment  of  segmented  objects  with  templates,  and  coding  of  video  sequences. 

We  see  affine  reconstruction  as  a  possible  alternative  to  tomographic  reconstruction  that  is  attractive  implemen- 
tationally  due  to  viewpoint  invariance  [Sha96],  lack  of  camera  calibration  requirements  [Sha96,  Moo96],  and  tolerance 
of  camera  translation  [Moo96].  More  detailed  discussion  is  presented  in  [Sch97b]. 

Shashua  and  Navab  analyzed  interpolation  and  extrapolation  errors.  Given  three  views  ,  and  aj  +  j ,  ex¬ 

trapolation  based  on  the  (i-l)-th  and  i-th  views  was  employed  in  reconstructing  an  approximation  to  the  (i+l)-th  view. 
Error  was  measured  as  the  mean  squared  distance  between  corresponding  points  in  the  extrapolated  and  actual  views. 
As  expected,  the  upper  bound  on  error  was  provided  by  extrapolation,  whose  error  decreased  to  1.1  ±  0.98  pixels  as 
the  separation  between  the  i-th  and  (I+l)-th  views  increased,  up  to  the  visibility  limit.  This  is  due  to  the  fact  that  ex¬ 
trapolation  is  usually  more  erroneous  than  interpolation,  given  identical  continuity  assumptions.  Scene  reconstruction 
error  was  measured  as  the  mean-squared  difference  between  predicted  and  measured  object  elevation.  A  cube  having 
greyscale  patterns  on  its  faces  was  employed  as  a  test  object.  The  average  depth  error  was  0.23  ±  0.3 1  percent  of  mean 
range  distance  r  to  target,  with  a  peak-to-peak  error  range  of  0.002r  to  0.007r. 

Practical  considerations  in  affine-based  3-D  modeling  for  MLI  applications  emphasize  the  accuracy  with  which 
the  various  points  can  be  located  in  terms  of  world  or  relative  coordinates.  In  imagery,  pointing  accuracy  is  limited 
by  focal  plane  quantization  error.  The  accurate  determination  of  sensor  position  and  orientation  is  crucial  to  establish¬ 
ing  the  camera  origin  points.  If  the  separation  between  sensor  origins  can  be  determined  accurately,  then  Shashua’ s 
method  [Sha96]  can  be  modified  to  yield  displacement-based  measures  of  relative  affine  structure,  per  [Moo96]. 

1.2.3.  Coregistration-based  techniques.  Another  approach  to  multi-look  imaging  involves  coregistration  of  the  multi¬ 
ple  looks  across  projections  (views)  and  spectral  bands  (in  the  case  of  multispectral  imagery).  If  the  spectral  bands  are 
not  severely  decorrelated  temporally,  current  coregistration  technology  may  be  adequate  for  inter-band  alignment 
within  ±1  pixel.  However,  each  projection  (taken  at  a  different  look  angle  and  altitude)  has  scaling,  rotation,  shift, 
and  anisomorphic  geometric  distortions  that  tend  to  make  registration  across  all  views  difficult  for  wide-angle  MLI. 
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Yeshurun  and  Schwartz  first  proposed  cepstral  filtering  for  region-based  solution  of  the  stereo  correspondence 
problem  [Yes89],  which  instantiates  a  correspondence  problem.  This  scheme  uses  an  adaptive  prefilter  followed  by 
autocorrelation,  which  provii  :s  prewhitening  that  reduces  the  noise  sensitivity  of  the  resulting  stereo  disparity.  Given 
a  stereo  pair  of  MxN-pixel  images,  the  stereo  image  formed  as  c  =  (a,b)  is  transformed  via  application  of  the  cepstrum 
to  yield  the  disparity  image  d,  as  follows: 

d  =  7(log[|7(a)  +  l|])  , 

where  ^denotes  the  Fourier  transformation.  The  difference  between  the  coordinates  of  the  origin  and  largest  peak  of 
d  yields  the  stereo  disparity.  Since  the  cepstral  approach  can  be  applied  to  small  neighborhoods  of  an  image  that  con¬ 
tain  corresponding  points,  it  is  useful  in  principle  for  imagery  that  has  space-variant  distortion.  Unfortunately,  the  lo¬ 
cation  of  correspondence  neighborhoods  is  itself  an  instance  of  the  correspondence  problem  that  may  simplify  the 
problem  implementationally  via  a  hierarchical  registration  scheme,  but  does  not  fundamentally  reduce  complexity. 

Ludwig  et  al.  [Lud94]  performed  an  error  analysis  of  Yeshurun  and  Schwartz’  technique,  to  determine  the  perfor¬ 
mance  of  cepstral-based  correspondence  as  a  function  of  image  noise  and  distortion.  It  was  suggested  that  LoG  win¬ 
dowing  functions  be  employed  for  vignetting  each  region  to  which  the  cepstrum  is  applied.  This  reduces  disparity 
artifacts  slightly  without  improving  algorithm  robustness  in  the  presence  of  scaling  distortion  or  photometric  variance. 

Smith  and  Nandhakumar  [Smi96]  employ  the  cepstrum  as  a  nonlinear  correlator  to  achieve  increased  tolerance  of 
noise  and  anisomorphic  geometric  distortions.  Their  enhancements  to  Yeshurun  and  Schwartz’  technique  (a)  replace 
the  peak  detection  process  in  the  power  cepstrum  with  a  more  robust  search  mechanism  in  disparity  space  and  (b)  cor¬ 
rect  for  foreshortening  effects.  The  result  is  a  reduction  in  sensitivity  of  the  corresponding  disparity  measure  to  pho¬ 
tometric  variation  and  anisomorphic  distortions. 

Given  the  foregoing  techniques,  advantages  of  coregistering  MLI  images  include: 

•  Effects  of  random  sensor  noise  can  be  ameliorated  somewhat  by  accumulation  of  the  coregistered  images,  and 

•  One  may  be  able  to  obtain  depth  information  for  common  points  across  all  views  of  an  MLI  image  sequence, 
for  purposes  of  3-D  model  construction. 

Unfortunately,  the  following  practical  disadvantages  can  accrue  from  erroneous  statistical  assumptions,  computational 
error,  and  continuity  assumptions  required  for  coregistering  multiple  views: 

•  Lateral  spatial  alignment  (coregistration)  error  may  induce  elevation  errors  in  stereo  reconstruction  of  depth 
maps  from  MLI  sequence  pairs,  which  errors  may  exceed  salient  target  feature  height;  and 

•  Computational  error  inherent  in  the  resampling  process  required  to  rectify  (coregister)  imagery  to  compensate 
disparity  effects  may  also  induce  elevation  errors  in  the  reconstructed  3-D  target  model. 

Although  region-based  cepstral  correspondence  matching  is  currently  our  method  of  choice  for  image  coregistration, 
additional  research  remains  in  global  coregistration  techniques.  Further  discussion  and  error  analyses  are  given  in 
[Smi96]  and  [Yes89], 
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1.2.4.  Comparison  of  3-D  models  and  2-D  images.  Given  an  image  a,  an  object  recognition  process  typically  extracts 
key  features,  then  seeks  a  model  M  that  best  matches  the  closest  view  of  the  feature  nsemble  [Fis81,Hut90,UI191], 
Image-model  alignment  can  be  inexact  due  (for  example)  to  input  noise,  model  specification  errors,  and  deformation 
in  nonrigid  objects.  Thus,  robust  assessment  of  alignment  accuracy  is  required.  The  common  assumption  of  a  Euclid¬ 
ean  distance  (called  the  image  metric  [Bas96])  between  image  feature  points  and  their  corresponding  points  in  the 
model  view  of  an  object  that  best  matches  imagery  assumes  that  images  (being  inherently  more  erroneous  than  mod¬ 
els)  induce  alignment  perturbations  that  should  be  measured  in  the  image  plane.  Although  this  assumption  may  be 
suitable  for  recognition,  object  classification  tends  to  benefit  more  from  minimal  object  deformation  that  tolerates  un¬ 
certainties  in  a  test  object’s  structure. 

Measures  that  compare  3-D  models  and  2-D  images  should  be  metrical ,  which  Basri  [Bas96]  defines  as  increasing 
monotonically  with  the  difference  between  M  and  a.  To  achieve  this,  the  class  of  transformations  applied  to  the  object 
are  customarily  extended  from  rigid  to  affine  transforms.  Although  this  bounds  the  rigid  measure  from  below,  an  up¬ 
per  bound  is  not  attained.  Previous  methods  tended  to  achieve  suboptimal  distances  or  did  not  produce  a  metric. 

Basri  and  Weinshall  [Bas96]  reported  a  distance  measure/that  compares  a  3-D  model  M  with  a  2-D  image  a  in 
closed  form.  This  penalizes  nonrigidity  incurred  by  an  optimal  affine  transformation  T  that  aligns  M  with  a  under 
weak-perspective  projection,  as  follows: 

A 

f(M,  a)  =  ^€'t  || T(M,  a)  -  R(M,  a)||z  , 

where  TRg  denotes  the  set  of  all  rigid  transformations  and  the  norm  is  denoted  by  (II II).  Thus/,  which  bounds  the  least- 
squares  distance  from  above  and  below,  facilitates 

1.  Direct  assessment  of  similarity  between  3-D  models  and  2-D  images; 

2.  Obtaining  upper  and  lower  bounds  on  the  image  metric  for  object  classification;  and 

3.  Derivation  of  an  initial  value  set  for  numerical  surface  fitting  procedures. 

An  additional  application  is  he  evaluation  of  hypothesized  correspondences  in  alignment  algorithms,  which  evaluate 
the  similarity  between  models  and  images  based  on  sparse  point-to-point  correspondences.  Although  this  technique 
can  produce  polynomial-time  matching  in  the  presence  of  occlusion  [Gri92],  matching  errors  invariably  result  due  to 
sparse  sampling,  which  can  be  mitigated  with  additional  correspondences  [Fis81].  The  measure/can  be  further  em¬ 
ployed  to  evaluate  the  amount  of  affine  distortion  applied  to  an  object  relative  to  a  prototype,  thereby  determining  the 
best-match  object  class  or,  in  certain  cases,  object  identity. 

2.  Kev  Issues  and  Assumptions 

We  assume  that  there  exist  the  following  three  primary  scenarios  for  MLI: 

1.  Narrow-angle,  Few  Looks  ~  the  sensor  look  angle  is  perturbed  only  slightly  from  a  reference  orientation 
(customarily  nadir  viewing)  and  a  few  (e.g.,  5  to  10)  images  are  taken  over  a  relatively  narrow  field-of-view 
(e.g.,  10  to  20  degrees). 
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2.  Wide-angle,  On-track  ~  the  sensor  moves  along  an  approximately  linear  or  curvilinear  track,  and  acquires 
imagery  over  few  c\  many  looks,  over  a  wide  field  of  view. 

3.  Loitering  —  the  target  is  reconnoitered  by  an  airborne  sensor  describing  a  simple  or  complex  curvilinear  path, 
with  unequally  spaced  views  from  different  elevations. 

Scenario  1  has  the  potential  of  producing  approximately  equally  spaced  looks,  some  of  which  can  be  corrupted  by  cov¬ 
er  interposed  between  sensor  and  target.  This  is  the  scheme  proposed  in  [PAR97],  upon  which  tomographic  recon¬ 
struction  of  the  target  would  be  based.  Scenarios  2  and  3  would  typically  acquire  unevenly  spaced  images,  with  scale 
differences  likely  as  sensor  altitude  (and,  possibly,  camera  magnification)  changes  unpredictably.  We  have  found  that 
such  schemes  are  often  amenable  to  3D  target  reconstruction  via  affine  structure  extraction  or  surface  fitting.  Tomog¬ 
raphy-based  approaches  would  be  confounded  by  large  partitions  of  missing  information  in  the  tomographic  recon¬ 
struction  space,  especially  if  few  looks  were  taken. 

3.  Methodology 

We  begin  with  a  one-dimensional  model,  for  purposes  of  illustration,  then  progress  to  a  two-dimensional  model 
of  cover,  which  is  instantiated  in  computer  software. 

3.1.  Theory  of  Multi-Look  Imaging 

Assumption  ! .  Let  a  spatial  domain  X  c  R  represent  (for  example)  a  background  region  viewed  by  an  imaging  sensor 
at  altitude  as ,  as  shown  schematically  in  Figure  1.  Let  opaque  cover  located  at  altitude  a  be  comprised  of  segments 
having  width  w. ,  where  1  <  i  <  n  .  The  segments  are  separated  by  horizontal  apertures  gj  that  have  centroids  at 

xC;  e  X.  Per  Figure  1,  we  assume  that  the  i-th  aperture  precedes  the  i-th  cover  segment  as  one  proceeds  away  from 
the  sensor. 


Lemma  1 ,  Given  Assumption  1,  if  the  sensor  images  a  contiguous  partition  [x™in,  x™ax]  of  X  through  aperture  ,  then 
the  following  statements  hold: 


1.  Look  angle  extrema  §™in (i)  =  tan 


-] 


1 

I  Sj  +  W 

is. J _ 


and  =  tan  1 


(  i-\  \ 

Si+  X  *j  +  wj 
_ L=J 


o  n  *  w/n  x  j  wax  ,,max . 

2.  Partition  extrema  x*  =  as  •  tan (<()v  )  and  x.  =  as  ■  tan((J)v  ) ;  and 

0  r.  ,  ,  r  •  max  min 

3.  Field  of  view  v  ■  =  x  ~x 

j  iii 

Proof.  The  proof  follows  directly  from  the  geometry  of  Figure  1.  ■ 
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The  preceding  model  does  not  suffer  loss  of  generality,  since  if  all  indices  i  are  in  the  interval  [l,n],  then  w-  =  0 
depicts  the  case  of  no  cover  and  describes  the  situation  where  there  are  no  apertures.  Alternatively,  if  a  given  aperture 
subtends  X,  then  there  is  no  cover  but  viewing  is  unobstructed.  In  practice,  the  model  of  Figure  1  is  best  applied  when 
a  target  is  situated  against  the  background  and  the  cover  segments  wi  are  vertically  thin  in  relation  to  the  sensor  or 
cover  altitude.  We  also  assume  that  the  apertures  are  sufficiently  large  to  render  diffraction  effects  negligible. 

Additionally,  the  preceding  one-dimensional  model  can  be  generalized  to  higher-dimensional  domains  by 
observing  that,  in  multi-look  imaging,  the  received  image  a  is  comprised  of  more  than  one  projection  of  the  target 
restricted  to  the  corresponding  projection  of  the  cover.  Since  the  cover  is  assumed  to  be  opaque,  application  of 
restriction  in  this  context  docs  not  compromise  rigor.  Hence,  the  one-dimensional  model  and  its  generalization  in  two 
or  more  dimensions  are  sufficiently  general  and  rigorous  to  portray  physical  reality  in  this  preliminary  study. 


Figure  1.  Schematic  sensor  geometry  for  multi-look  imaging  through 
perforated  cover,  where  *  denotes  lateral  sensor  position. 

2  X 

Assumption  2.  Let  domain  XcR  and  assume  that  cover  is  specified  by  an  image  ceR  located  ac  meters  (m) 

x 

above  a  target  whose  elevation  map  (or  equivalent  projection)  is  denoted  by  the  Boolean  image  be  B  .In  this  study, 

we  adopt  the  convention  that  the  values  of  c  denote  the  transmission  coefficients  of  the  cover.  Let  an  imaging  sensor 

of  focal  length /and  magnification  M  be  located  as  meters  above  the  target  plane,  as  shown  in  Figure  2.  For  each 

x 

configuration  of  cover  c,  and  target  bz,  where  1  <  i  <  n  ,  let  the  sensor  form  an  image  a(.eF  .  Further  assume  that 
the  transmissive  medium  between  target  and  cover  (cover  and  sensor)  has  absorption  and  scattering  coefficients 

aw  htc  ( acs ’  bcs)  and  v°lume  scattering  function  Pfc  (PC5). 

Observation.  When  simulating  the  formation  of  sensor  image  a,  one  of  two  cases  pertain: 

Case  1:  A  flat  target  (or  one  that  can  be  assumed  to  be  flat),  where  b  denotes  a  reflectance  map,  or 

Case  2:  3-D  target,  where  b  denotes  an  elevation  map. 
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For  purposes  of  illustration,  the  following  development  is  specific  to  the  first  case,  and  the  second  case  is  discussed 
beginning  with  Theorem  3. 

Definition  1,  A  spatial  transformation  f  :  Y  — » X  is  a  map  between  spatial  domains.  Given  an  image  a  €  ,  a 

Y 

spatially  transformed  image  be  F  is  denoted  by 

b  =  a  o  /  =  {(y,  a(y))  :  ye  Y}  . 

IheoremJ,  Given  Assumption  1  and  the  geometry  of  Figure  2,  the  projection  S  of  a  point  x  e  domain(c)  to  a  point 
ye  domain(d)  is  given  by 

X  =  S( y)  =  f— - ■  h'  ■  (sinev,  cos0v)  , 

\  a5  J 

where  W  =  (lp\(y)]2  +  [P2(y)]2)lt2  and  0y  =  ^nl(p2(yVP\(y))y  vv/r/z  pk{ y)  denoting  projection  to  the  k-th  coordinate , 
which  implies  that  d  =  co  S . 

Proof.  From  the  similar  triangles  of  Figures  1  and  2,  the  target  dimension  along  the*-  and  y  -axis  of  domain(b) 
is  scaled  by  ( as-ac)/as  in  domain( c).  This  scale  factor  is  also  applied  to  the  hypotenuse  hf  to  yield  h  in 
domain(c).  The  hypotenuse  h'  is  projected  via  angle  0y  which  does  not  vary  between  b  and  c.  Hence,  we  have 
the  expression  S(y)  =  ((as  -ac)/as)  •  W  •  (sin0v,  cos6v) ,  and  d  =  co  S  follows  from  Definition  1 .  ■ 

Z 


Figure  2.  Three-dimensional  sensing  geometry. 

Theorem  2.  Given  a  volume  scattering  function  P  :  (-ju,  jt  ]  -»  [  0,  1  ]  and  a  scattering  distance  d  e  R+  then  a  tem¬ 
plate  s  :  R  X  X  — >  Rx  that  instantiates  a  single-scattering,  near-field  approximation  of  P(t>)  is  defined  in  terms  of 
its  weights  as 
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Sy(d)(x)~  P(d),  where  $  =  tan  . 

Proof.  Given  the  geometry  of  Figure  3,  it  suffices  to  state  that  i3  =  tan_1(||y-x||/cf) .  ■ 

3.2.  MLI  Model 

We  next  present  several  algorithms  and  theorems  that  we  are  currently  instantiating  in  a  computer  model,  which 
would  simulate  the  effect  of  multiple  views  of  a  target  object  through  perforated  cover. 

Algorithm  L  Given  Assumption  1,  if  b  denotes  a  greyscale  image  (e.g.,  a  reflectance  map),  then  the  formation  of  im¬ 
age  a,  can  be  simulated  by  performing  the  following  steps: 

Step  1  ■  Form  the  spatial  !ransform  S  -  :  X  — >  X ,  from  which  the  projection  of  c,-  to  the  target  plane  is  given  by 

X 

c  0  5 ,  per  Definition  1 .  From  5,  sensor  geometry,  and  the  geometry  of  X,  one  obtains  a  map  d  e  (R+) 
of  the  propagation  distance  from  the  target  plane  to  the  sensor  as  a  function  of  a  point  x  €  X ,  as  shown 
in  Theorem  2. 

SteP  ?•  Given  the  media  optical  parameters  atc ,  b,c ,  and  Pfc ,  form  the  attenuation  template  s,c  :  R+  x  X  ->  RX , 
which  is  parameterized  by  the  propagation  distance  between  target  and  cover,  per  Theorem  2.  An  addi¬ 
tional  attenuation  template  s„  is  formed  symmetrically  from  acs ,  bcs ,  and  pCJ  that  portrays  attenuation 
in  the  propagation  path  segment  from  cover  to  sensor. 

3'  Apply  S,  stc,  and  s„  to  c  to  yield  s,-  =  s,c(d()  ®  s„(d,)  and  an  approximation  to  a,-,  as  follows: 

a,-  =  [b  *  (c,  o  Si)]  ®  Si ,  (II) 

The  preceding  equation  does  not  include  realistic  receiver  effects,  such  as  noise,  nonuniform  pixel  gain,  or  missing 
pixels,  which  we  have  discussed  in  [Sch96a],  and  which  would  be  included  in  the  completed  model. 


X  H 


Scattering  Path 


*) 


-►  X 


Scattering  Center 


Figure  3.  Linear  approximation  for  single-scattering  geometry. 

Theorem  3.  Let  a  cylindrical  target  of  radius  r  and  height  h  be  viewed  by  an  ideal  camera  of  unitary  magnification, 
at  look  angle  <t>  measured  away  from  the  vertical).  The  apparent  target  dimension  is  given  by  h  ■  cos<| >  +  r  ■  sin<]) . 
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Proof.  The  proof  follows  directly  from  the  construction  of  Figure  4a. 


Figure  4.  Three-dimensional  target  (a)  geometry,  and  (b)  sensing  configuration. 

Theorem  4.  Given  the  geometry  of  Figure  4b,  if  P  denotes  a  perspective  correction  matrix;  C,  R,  and  G  denote  camera, 

rotation,  and  translation  matrices ,  respectively;  and  Mh  denotes  local  target  coordinates;  then  the  target  coordinates 

T 

projected  to  the  focal  plane  of  a  sensor  at  altitude  as  are  given  by  C  =  (x,  y,  z,  1 )  ,  such  that  Cf  =  P  C  R  G  Mh . 
Proof  Let  Mh  denote  loc.il  target  coordinates,  and  let  translation  matrices  G  and  C  be  given  by 


(  1  OO-^pf 

f  1  0  0 -^(pf 

G  = 

0  1  0  -p2{ p) 

and  C  = 

0  1  0  -p2( p) 

0  0  1  -p3(p) 

0  0  1  -p3(p) 

<000  1  j 

<000  1  J 

where  r  denotes  the  coordinates  of  the  sensor  camera’s  focal  point,  with  the  camera  lens  nodal  point  located  at  p. 
Let  R  =  R^Rq  be  a  combination  of  two  rotations  R^  (tilt  angle)  and  R0  (pan  angle),  such  that 

(  cos  0  sin  0  0  0  ^ 

_  -sin  0  cos  $  cos  0  cos  §  sin  <J)  0 
sin  0  sin  $  -cos  0  sin  (J)  cos  (j)  0 
^  0  0  0  1  ; 

Define  the  perspective  projection  matrix  P  as  follows: 

'  1  0  0  0  ^ 

P  _  0  1  0  0 

0  0  1  0 

v  0  0  -1/f  1  > 

where  f  denotes  the  sensor  focal  length.  Assume  that  translation  and  rotation  matrices  T  and  R  are  given  by 
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'  1  0  0  -Xm  1  1  0  0  xm  1  1  ^  0  -xm  ' 

T  =  010  ->'m  t_1  =  0  1  0  yD  and  R  =  °  C0S  ^  sin  5  "ym 

0  0  1  -zm  0  0  1  zm  0  -sin  C,  cos  £  -zm 

,oooiJ  toooij  V00  0  1, 

From  the  expression  of  Theorem  3  in  two  dimensions,  and  letting  r  and  h  denote  the  radius  of  height  of  a  cylin¬ 
drical  target  inclined  at  an  angle  £  from  the  horizontal,  we  have  that 
Mh  =  (/i sinc{)  +  2rcos(|),  /icos<|)  +  2rsin<(),  h§) ,  where  (J)  e  [0,  2n)  and  |/z0|</z/2.  One  thus  obtains 

wh  =  T-1R  T  Mh  , 

which  yields  the  projection  of  target  coordinates  to  image  space  as 

C  =  PCRGMh.  ■ 


of  Tomographic  Reconstruction 


Computerized  axial  tomography  customarily  projects  a  planar  beam  of  X-rays  through  an  object,  where  the  trans¬ 
mitted  X-rays  are  measured  by  a  linear  array  of  detectors.  By  rotating  the  transmitter/detector  through  180  degrees  at 
small,  fixed  angular  increments  then  applying  a  reconstruction  algorithm,  a  high-resolution  of  a  given  cross-section 
through  an  object  may  be  obtained.  Combining  these  cross-sections  into  a  solid  model  yields  3-D  information  con¬ 
cerning  the  object  of  regard.  In  this  study,  we  concentrate  on  airborne  MLI  imagery  of  battlefield  scenes,  and  inves¬ 
tigate  whether  or  not  tomographic  imaging  techniques  could  be  applied  to  such  imagery.  We  call  this  process 
battlefield  tomography ,  and  note  that  control  of  the  sensing  parameters  or  knowledge  of  target/media  properties  would 
not  necessarily  be  precise.  In  contrast,  medical  tomography  has  the  advantages  of  (a)  well-known  optical  parameters 
of  various  tissues  that  vary  with  relative  predictability  cross-population  and  among  various  pathological  conditions, 
(b)  a  highly  constrained  geometry  where  transmitter  and  detector  positions  are  known  accurately,  (c)  quantization  bin- 
size  that  can  be  set  sufficiently  small  to  meet  diagnostic  accuracy  constraints,  and  (d)  offline  processing  capability  that 
admits  reasonable  computational  cost  to  achieve  high  reconstruction  accuracy. 

Assume  that  a  source  and  detector  assembly  is  used  to  sense  transmittance  at  each  linear  increment  along  the  com¬ 
mon  z  axis  of  superimposed  3-D  Cartesian  and  polar  coordinate  spaces.  Given  a  two-dimensional  domain  X,  one  thus 
obtains  a  collection  of  images  a  =  { (i,  a,)  :  a,  €  Rx } .  Given  incident  beam  intensity  IQ ,  a  can  be  transformed  to  ob¬ 
tain  an  in-plane  density  function 


d(p,  0) 


a(p,  0)  ’ 


/  2  2 
p  =  a/x  +y 


and  0  =  tan  (y/x) 


(x,  y)  e  X  , 


and  the  beam  direction  forms  angle  0  with  the  y-axis.  We  assume  for  purposes  of  convenience  that  d  has  the  same 
set-theoretic  form  as  a  and  can  be  operated  upon  by  the  Radon  transform  to  yield  the  following  expression  for  the  pro¬ 
jection  process: 
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b(f:,  0)  =  J  J  d(p,  0)  ■  8(x  cos  9  +  y  sin  0)  dx  dy,  (x,  y)  e  X  , 

— oo  — oo 

where  8  denotes  the  Dirac  delta  function  with  p  and  0  defined  previously. 

In  the  case  of  CAT  scanning,  image  reconstruction  is  based  upon  the  similarity  properties  of  the  two-dimensional 
X  X 

Fourier  transform  jt  :  R  — >  R  Given  c  =  ^7"(d)  we  obtain 

J(b(p,0))  =  c(5,0)  , 

1  2  2 

where  J;  =  Vu  +  v  and  (u,  v)  €  domain(^F( b))  Thus,  each  density  function  yields  a  function  c,  that  represents 
a  radial  cross-section  through  the  2-D  FT  of  the  object.  By  sampling  0  in  small  steps,  the  resulting  transforms  can  be 
interpolated  to  approximate  c,  which  can  be  inverse-transformed  to  yield  a  further  approximation  to  b.  Performed  over 
a  range  of  linear  increments  along  the  z-axis,  this  process  yields  an  approximation  to  the  3-D  X-ray  density  map  of  the 
object. 

Assuming  (for  purposes  of  brevity)  that  c  is  defined  on  domain  X,  the  computational  efficiency  of  the  FT-based 
reconstruction  algorithm  can  be  increased  at  the  expense  of  accuracy  by  employing  the  back-projection  algorithm , 
which  is  given  by 

n 

c(x,  y)  =  Jb(x  cos  0  +  y  sin  0,  0)  d0,  (x,  y)  €  X  .  (Ill) 

0 

Each  projected  density  function  is  projected  back  (expanded)  along  the  axis  of  the  beam  at  orientation  0  to  form  a  2- 
D  image  that  is  comprised  of  1-D  images  parallel  to  the  scanning  axis.  By  summing  (superposing)  all  such  images 
over  one  z-axis  bin,  an  approximation  of  the  cross-sectional  view  can  be  obtained  at  that  bin. 

In  practice,  the  reconstructed  function  c  is  comprised  of  d  blurred  by  a  point-spread  function  of  form 

2  2  1/2 

f(x,  y)  =  (x  +y  )  .  By  highpass  filtering  each  1-D  projection  in  b  prior  to  backprojection  with  a  filter  whose 

MTF  increases  linearly  with  spatial  frequency,  one  can  approximate  the  inverse  Radon  transform,  from  which  d  is  ob¬ 
tained.  Lowpass  filtering  of  the  reconstruction  d  reduces  the  appearance  of  random  noise  at  the  expense  of  resolution. 

4.  Error  Analysis  and  Physical  Model  Results 

Three  types  of  error  are  expected  to  predominate  in  tomography-based  reconstruction  algorithms.  First,  occlusion 
of  the  target  by  opaque  or  translucent  cover  segments  tends  to  reduce  the  information  about  target  reflectance  (UV  and 
visible  wavelengths)  and  induced  or  inherent  luminosity  (UV  or  infrared  wavelengths').  This  would  locally  decrease 
the  signal-to-noise  ratio  (SNR)  at  a  target  point  visible  in  a  given  view  of  the  object.  In  the  presence  of  point-spread 
function  (PSF)  effects,  total  occlusion  at  given  target  points  could  yield  a  small  amount  of  useful  information  due  to 
channel  crosstalk,  provided  that  the  PSF  could  be  accurately  estimated  and  deconvolved  from  the  received  image. 
However,  this  is  unlikely  in  battlefield  practice  without  clearly  visible  reference  objects  (e.g.,  manufactured  targets 
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such  as  vehicles)  from  which  one  can  estimate  edge-spread  effects  [Sch95,Yan95,Sch96b].  In  Section  4.1,  we  discuss 
this  issue  in  the  context  of  an  information-theoretic  model  of  target  occlusion. 

A  second  source  of  error  in  target  reconstruction  from  multiple  views  occurs  during  projection  of  one  or  more 
pixels  of  a  discrete  image  from  the  sensor  focal  plane  to  a  target  point.  The  discrete  focal  plane  has  quantization  error 
that  is  magnified  by  the  projection  process.  Additionally,  media  effects  such  as  scattering  and  attenuation  reduce  the 
certainty  with  which  a  point  may  be  located  in  the  focal  plane.  The  combined  first-order  effects  of  focal  plane  quan¬ 
tization  and  PSF  effects  yield  spatial  errors  that  are  projected  to  the  domain  of  the  reconstructed  target.  For  example, 
in  near-nadir  viewing  at  low  altitudes,  such  errors  tend  to  be  minimized  when  paraxial  approximations  are  justified  by 
the  sensor  configuration.  In  contrast,  given  wide-field  imaging  at  long  range  distance  (e.g.,  a  cruise  missile  viewing 
a  distant  target  on  the  edge  of  a  clearing,  as  the  missile  approaches  above  treetop  level),  slant-path  effects  due  to  natural 
or  manmade  obscurants  can  severely  degrade  the  received  image.  If  targets  are  poorly  resolved,  then  quantization  error 
may  predominate.  The  geometric  basis  for  such  effects  is  summarized  in  Section  4.2  and  [Sch97b]. 

Target  reconstruction  algorithms  present  further  problems.  For  example,  tomography  based  on  the  Fourier  trans¬ 
form  (per  Section  3.3)  customarily  requires  specially  designed  filtering  or  data  windowing  techniques  to  maximize 
SNR  and  minimize  artifacts  such  as  high-frequency  ringing  or  vignetting  effects  [Nat86].  Additionally,  the  FT  has 
well-known  errors  that  result  from  division  by  small-magnitude  coefficients,  aliasing  of  high  frequency  information, 
etc.  The  backprojection  algorithm  given  in  Equation  (HI)  has  associated  errors  of  approximation,  as  well  as  compu¬ 
tational  errors  inherent  in  the  transcendental  functions,  which  tend  to  be  dominated  by  the  key  error  of  angular  quan¬ 
tization  binsize.  In  fairness,  we  note  that  such  errors  are  not  absent  from  geometric  reconstruction  algorithms,  for 
example,  the  relative  geometry  approach  of  Shashua  and  Navab  [Sha96].  A  key  problem  in  geometry-based  recon¬ 
struction  is  accurate  location  of  the  viewpoint  and  target  reference  points.  Also,  solutions  to  the  correspondence  prob¬ 
lem  are  required  that  have  error  approaching  the  Nyquist  limit.  Errors  involved  in  such  processes  are  discussed 
extensively  in  [Nat86,  Yes  89  Smi96]  and  [Sch97b]. 

Section  4.3  contains  a  summary  of  a  computer  simulation  model  of  MLI  processes  that  was  developed  during  the 
study  and  is  being  enhanced.  An  additional  topic  investigated  in  this  study  is  overviewed  in  Section  4.4,  namely,  phys¬ 
ical  models  that  can  be  employed  in  place  of  the  computer  model  for  fast  MLI  simulation  using  detailed  target  and 
background  environments.  The  physical  modeling  methods,  which  have  long  been  known,  suffer  from  lack  of  realism 
at  high  resolution,  but  tend  to  produce  useful  imagery  for  early  algorithm  testing.  Fuithermore,  the  model  configura¬ 
tion  can  be  easily  varied,  and  results  can  be  verified  directly  by  physical  measurement  of  the  sensor  parameters. 

4.1.  Information-Theoretic  Model  of  Target  Occlusion 

In  this  section,  we  generalize  the  model  developed  in  Section  3  to  produce  an  estimate  of  information  contained 
in  a  given  projection  of  an  obscured  target  viewed  through  perforated  cover. 

Definition  2.  If  a  process  P  has  n  equiprobable  outcomes,  then  the  entropy  of  P  is  given  by  H( P)  =  log(n),  where  the 
logarithm  is  taken  to  the  base  two. 
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Definition  3.  If  a  process  P  has  n  possible  independent  outcomes,  each  of  which  has  probability  p . ,  then  the  entropy 
of  P  is  given  by 

n 

m P)  =  I  -p,  •  iog(p,) . 

i  =  1 

If  the  outcomes  of  P  are  not  statistically  independent  (which  occurs  in  practice),  then  the  equality  relation  in  the  pre¬ 
ceding  equation  is  replaced  with  the  inequality  (< ). 

Assumption  3,  Let  the  probability  of  viewing  through  cover  (per  Figure  1)  be  expressed  in  terms  of  an  image  c  that  is 
comprised  of  component  probability  maps  c(  e  [0,  1  ]  .  Let  the  component  images  of  c  be  coregistered  on  a  common 
domain  YdX,  such  that  the  n  domain  points  in  the  target  plane  to  which  a  given  focal-plane  point  x  e  X  could 

project  correspond  to  one  and  only  one  point  in  Y.  We  adopt  the  convention  that,  in  the  z'-th  view,  if  a  pixel  c((x)  is 
zero-valued,  then  there  is  no  view  through  the  cover  at  that  location.  Otherwise,  there  is  partial  (total)  information 

transmission  from  the  underlying  image  if  0<c((x)  <1  (c,(x)  =1).  Let  us  conceptualize  MLI  imagery  as  comprised 
X 

of  multiple  views  ai  e  F  ,  where  \ <i<n ,  which  are  coregistered  on  Y  =  domain{ c)  to  produce  an  image  b  that  is 

y 

comprised  of  images  b/  e  F  each  of  which  corresponds  to  a  view  in  a.  Let  X  denote  a  flag  value  and  let  the  operation 

/  :  [0,  1]  (0,  1  ]  u  {X}  map  nonzero  values  to  themselves  and  zero  to  X.  Further  assume  that  the  operation 

LRuUR  is  a  refinement  of  the  logarithm,  such  that 

h(X)  =  ( losw  if  *€  (0-  n 

[  0  otherwise 

Theorem  5.  Given  Assumption  3,  the  entropy  of  c  is  given  by 

n 

H(c(y))<  X-*i(y)-*(/[c,-(y)])  • 

/=  1 

Proof.  The  proof  follows  directly  from  the  givens  and  Definition  3.  ■ 

Assuming,  as  before,  that  b  and  c  share  a  common  domain  Y,  then  H( c(y))  portrays  the  entropy  of  a  pixel 
b(y)  =  (b  j  (y),  b2(y),  b^y),  bfl(y)) ,  where  b;(y)  denotes  the  /-  th  coregistered  view  of  the  y-th  target  pixel. 

Observe  that  this  information  is  useful  for  target  recognition  purposes,  as  discussed  in  the  following  algorithms. 

Algorithm  2.  Given  Theorem  5  and  the  preceding  observation,  one  could  exclude  the  pixels  of  c  that  have  no  target 
information  via  the  operation 

f  =  x=0(c)  • 

Algorithm  3.  Given  Theorem  5  and  the  preceding  observation,  it  is  possible  to  determine  the  indices  of  projections 
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that  exceed  an  entropy  thresh.  >ld  T,  as  follows: 

n 

D  =  domain(x>T(Z  c())  . 

i'=  1 

Thus,  one  could  preselect  certain  views  for  likelihood  of  useful  information.  This  technique  also  suggests  the  use  of 
the  singular  value  decomposition  (SVD)  for  determining  significant  views. 

4.2.  Projective  Errors 

From  Figure  1,  the  projection  of  a  point  x  in  a  one-dimensional  domain  X  to  the  i-lh  focal-plane  pixel  of  a  sensor’s 
imaging  device  looking  horizontally  constitutes  the  worst  case,  since  the  target  is  in  the  focal  plane  periphery.  At  al¬ 
titude  a  ,  focal  length  f,  and  magnification  M,  the  projection  from  a  target  point  xeXto  the  focal  plane  is  given  by 


i  =  f  ■  cot Tm  ■  ^  -  tan  ^  . 

I  12  \aj\) 


Taking  the  derivative  of  i  with  respect  to  x,  we  obtain  the  worst-case  focal-plane  error  3i  that  results  from  a  location 
error  Ox  in  the  target  plane,  as  follows: 


ffl  +  cotf—  2  •  tan  1 

l  UL  W  J; 

2  2 
as+x 


Oi/Ox.  M  ~  1 


Si/ax,  M  =  2 


c.00‘4 

0.0012- 

OOOtf  ' 

0.«X®f 
0  00064 
o.ooo4 


x  <m) 


^  Zm 


eSo  $56  ifioo  '  1C0C  2000 

Sensor  Altitude  as  ,meters 

Figure  5.  Worst-case  target-to-camera  projection  error  (a)  M  =  1  and  (b)  M  =  2. 

Example.  Since  f  and  M  are  usually  fixed,  we  set  f  =  35mm  and  M  =  1,  which  are  reasonable  implementational  values. 
Varying  the  sensor  altitude  from  0.1km  to  1  km  and  varying  the  range  distance  x  from  100m  (nadir  viewing)  to  2km 
(slant-path  viewing),  we  obtain  the  result  shown  in  Figure  5a,  where  Oi/Ox  =  0.035 /as .  Setting  M  =  2,  we  obtain 
the  result  of  Figure  5b,  which  is  described  by  the  following  equation: 


as  •  I  1  +  cot!  2  *  tan 


2  2 
a.  +  x 
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The  minimum  error  scenario  occurs  when  the  target  is  on-axis,  i.e.,  when  the  sensor’s  optical  axis  coincides  with  the 
line  segment  between  x  and  as .  Such  error  is  described  by  the  expression 


3i  =  f  •  tan^M 


tan  \  —  |- tan 


-\(x  +  dx 


which  reduces  to  di  ~  5(x  •  f/(Mx))  in  the  paraxial  assumption. 

Example,  In  the  minimum-error  case  for  M  =  1,  the  paraxial  approximation  at  as  =  400m,  x  =  1km  ground  range,  and 
dx  -  ±1  m  ground  error  yields  a  focal-plane  error  of 


3i  =  9(x  •  f/(Mx))  =  ±35^  =  ±lm  ■  1  •  0.035m/103m  . 


Given  9  micron  pixels,  3i  =  ±35(i/9p  =  ±3.9  pixels  .  With  20  micron  pixels,  di  =  ±35|l/20|i  =  ±1.7  pixels. 

Under  similar  constraints,  during  target  recognition,  the  projection  of  a  ray  bundle  that  subtends  the  i-th  focal 
plane  pixel  to  a  target  point  x  e  X  is  given  by 


x 


The  general  expression  for  ground  error  as  a  function  of  focal-plane  error  (the  derivative  of  x  with  respect  to  i)  is  given 
as  follows: 


9x 

ai 


a  J  1  +  cot 


ftan  1  (i/f ) 


M 


M(f2  +  i2) 


(V) 


Figure  6.  Prol  ction  of  focal-plane  error  to  the  target  plane  at  (a)  M  =  I  and  (b)  M  =  2. 

Setting  f  =  35mm  and  M  =  1,  as  before,  and  varying  as  from  0.1km  to  1km  and  i  from  O.Olf  to  l.Of,  we  obtain  the 
result  shown  in  Figure  6a.  Within  the  paraxial  assumption,  the  preceding  equation  reduces  to  dx  ~  3i  •  Mx/f  .  Set- 
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ting  M  =  2,  we  obtain  the  following  result  shown  in  Figure  6b: 


f  •  as  •  1  +  cot 


tan  1  (i/f ) 


9i  Im  =  2  2(f2  +  i2) 

Example,  Given  the  preceding  sensor  parameters,  in  the  worst  case  at  unitary  magnification  (Reference  Figure  6a),  a 
focal -plane  error  of  ±15  (I  yields  a  spatial  error  in  the  target  plane  computed  from  Equation  (V)  as 


dx  =  di  ~ 
a  l 


=  ±0.075 m  =  ±15(1  •  -5  ■  10 


a.  =  400m 


In  practice,  we  currently  estimate  that  additional  effects  (e.g.,  atmospheric  PSF,  camera  noise,  reflections,  etc.)  could 
increase  3x/3i  by  a  factor  of  two  to  three,  thereby  rendering  3x  sufficiently  large  to  corrupt  high-resolution  target 
reconstruction.  Although  not  a  significant  factor  in  early  research  in  battlefield  MLI  (due  to  assumptions  of  low  spatial 
resolution),  such  errors  could  severely  degrade  reconstruction  of  higher-resolution  target  models. 

The  preceding  error  analysis  is  required  for  parameterizing  image  formation  and  reconstruction  processes.  In  the 
image  formation  step,  a  knowledge  of  3i/3x  is  required  due  to  sensor  positioning  and  aiming  errors  that  are  projected 
to  the  focal  plane.  During  the  selection  of  a  target  reconstruction  technique,  the  focal  plane  error  3i  directly  influences 
the  accuracy  of  target-plane  projection,  as  shown  above. 

We  next  discuss  projective  errors  inherent  in  the  three-dimensional  model  of  Figure  2,  as  well  as  practical  impli¬ 
cations  of  such  errors  for  low-altitude  airborne  sensing. 

Theorem  6.  Given  Equation  (IV)  and  the  geometry  of  Figure  2,  the  projection  S  of  a  point  x  e  domain(c)  to  a  point 


y  e  domain(d)  is  given  by 


ra  -a  \ 

x  =  S( y)  =  — J  •  hf  •  (sin0v,  cos0v)  , 


where  h '  -  ([pf  y)]2  +  [p2(yV2)112  and  0y  =  tenl(p2(yyp\(y))>  with  pk(  y)  denoting  projection  to  the  k-th  coordinate. 
Proof.  The  proof  follows  obviously  from  the  givens  and  spherical  trigonometry.  ■ 

From  Theorem  6  as  well  as  Figures  1  and  2,  the  projection  of  a  point  x  e  X  to  a  point  (i,j)  in  the  focal  plane  of  a  sensor 
(or  camera)  having  magnification  M,  focal  length  f,  and  an  anterior  nodal  point  at  as  is  given  by 


#  r* *  ,[(Jp\(x)i+p2(x)2 

(i,  j)  =  f  ■  tan  M  •  -  -  tan  - - 

2  a _ 

v  v  s 


(cos0,  sin0) 


where©  =  tan  (p2(x)/ p{(x)) . 


Error  analysis.  With  the  exception  of  0,  which  is  computed  from  x,  the  analysis  of  error  in  i  and  j  proceeds  similarly 
to  the  one-dimensional  case,  since  Equation  (VI)  merely  portrays  a  rotation  through  angle  0  of  the  projection  given  in 
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From  basic  trigonometric  identities.  Equation  (VI)  reduces  to 


f  *  tan 


(i.  j)  = 


/ 

r  ( 

M  * 

71  ♦  -! 
--tan 

\ 

/ 

L  V 

P  |(x)  +  P2(x) 


2\t 


a/1  +(/72(x)2/A’|(x)2) 


f  P2W\ 

l  >,(X)J  ' 


(VII) 


Letting  v  =  tan 
given  by 
3i 


r 

r  ( 

M 

n  -1 

-  -  tan 

2 

V 

V 

J/7,(X)2  +  /J2(X)2>'' 


f  /72(x)  V 


,  the  projective  error  components  of  i  in  Equation  (VII)  are 

fM  •  p,(x)  •  (1  +  v) 


P i(x)‘  • 


,  P2W 

1+ - 2 

V  PiW  ) 


1  + 


P\{x)  +  p2(x) 


H 


(p,(x)2  +  p2(x)2) 


J  l  p2(x) 

V  P|(x)2y 


and,  symmetrically  for  the  second  coordinate  of  x,  we  have 


di 


f  ■  p2W  v 


fM  ■  p2(x)  •  (1  +  v) 


dp2(x) 


P\W ■ 


/  .  ,2x3/2 

'  P2{x)  \  f 

1  +  — — 


v  P i(x)  ) 


1  + 


p,(x)  +P2(x) 


■  l(Pi(x)2  +  p2(x)2) 


I  + 


P2(x) 


V  P,(x)  J 


We  similarly  obtain  the  error  components  of  j  as 


dj 


a7Tfe  =  f  ^(X)' 


V-  p2(x) 


p,(*r  ■ 


1  + 


P2(x ) 

P,(x)^J 


2v3/2 


P\(x)p2{x )•  |1  +  — - - 

Pl(x) 


M  ■  (1  +  v  ) 


1  + 


Pl(x)  +P2(x) 


•  l(p,(x)2  +  p2(x)2)  ■ 


1  + 


P2(x) 


and 


P\(x)  J) 


3j  _ 


dp2(x) 


=  f  ■  p2(x)  ■ 


v  ■  p,(x) 


p,(x)'  • 


^  |  p2(x) 

V  /^(x)2,! 


P i(x)  •  p2(x)  •  1  + 


p2(x) 

Pl(x)2 


M  •  (1  +  v)  ■  p2(x) 


1  + 


P|(x)  +  p2(x) 


■  l(p,(x)2  +  p2(x)2) 


1  + 


p2(x) 


v  p,w 
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The  preceding  error  analyses  lead  to  two  important  observations,  namely:  (1)  the  forward  projection  errors  are 
stable  over  the  postulated  operational  range  400m  <  as  <  800m  and  600m  <  x  <  2km  ,  and  (2)  the  error  function 

graphed  in  Figure  2b  could  be  employed  in  constraining  selection  of  near-optimal  sensor  configuration  for  image 
acquisition  at  low  error  figure  in  a  variety  of  operation  scenarios. 

With  regard  to  observation  1),  above,  the  stability  of  a  given  projection  error  function  is  determined  from  the 
slope  of  the  error  surface  (e.g.,  as  shown  in  Figure  2b).  Only  in  the  range  as  <  200m  or  x  <  400m  is  projection  error 
significant.  Note  that  the  region  where  x  <  as  is  an  artifact  and  can  be  neglected  since  it  is  not  physically  feasible. 

Concerning  observation  2),  above,  within  the  operational  range  of  altitude  and  range-to-target  values  where 
x  >  as ,  we  observe  that  di  /  9x  is  sufficiently  smooth  to  be  encoded  either  as  a  low-order  polynomial  or  in  a  lookup 

table  of  sufficient  resolution.  Hence,  the  transcendental  operations  involved  in  the  computation  of  Equations  (I)  and 
(II)  could  be  avoided.  This  could  render  the  estimation  of  image  acquisition  error  feasible  on  fast  DSPs  or  small  par¬ 
allel  processors  with  few  floating-point  capabilities.  For  example,  the  trajectory  of  a  small  airborne  vehicle  could 
thus  be  constrained  in  real  time,  such  that  imaging  error  3i  /  3x  as  a  function  of  range  could  be  held  to  near-optimal 
values. 

In  contrast  to  the  preceding  observation,  note  that  Figure  3  illustrates  back-projection  error  (e.g.,  projection  of 
focal-plane  error  to  the  target  plane)  to  increase  with  decreasing  altitude  and  displacement  in  the  focal  plane.  In  fig¬ 
ure  3b,  the  surface  slope  is  nearly  linear  from  i  =  0.015  to  i  =  0.02,  for  500m  <  as  <  1km  .  Thus,  it  is  reasonable  to 
conclude  that,  by  merging  the  graphs  of  Figures  2b  and  3b,  one  can  infer  that  a  sensor  altitude  of  400m  <  as  <  600m 
for  3 <zs  <  x  <  1.5<2S  is  near  optimal  for  image  acquisition  and  target  reconstruction,  under  the  previously-mentioned 
sensor  configuration. 


It  is  often  useful  to  employ  a  back-projection  technique  for  error  analysis,  whereby  errors  in  the  camera’s  focal 
plane  or  optical  system  can  be  referred  to  the  target  plane.  This  is  especially  germane  to  target  reconstruction,  where 
it  may  be  desirable  to  determine  the  error  in  the  reconstructed  target  that  results  from  focal-plane  quantization  effects. 

From  Theorem  6  and  Figure  2,  the  projection  to  a  point  x  e  X  of  a  point  (i,j)  in  the  focal  plane  of  a  sensor  (or 
camera)  having  magnification  M,  focal  length  f,  and  an  anterior  nodal  point  at  as  is  given  by 


x  =  gs  •  tan 


[. 2  ~2\ 
a h  +J 

f  M 


(""(ns)  cos(nM 


(VIII) 


Error  Analysis.  Differentiating  p{(x)  and  p2(x) ,  x  e  X ,  with  respect  to  i  and  j,  we  obtain  the  following  expres¬ 
sions  for  error  in  a  two-dimensional  target  plane: 
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agiW  *» '  (|3m  '  "(A) + “"(A)  • (ji" h  j3)) 

8i  "  f  M V  JiL? 

apiw  ^•(JMi'c°s(rJM)-si'’(rii)  (i2-j3)) 

3j  ,  -.2.  fl  Tl 

}p2(x)  _  ^  (i3M  cos(rTi)-cos(rM)  (i'2+J3)) 

8i  ’  f-M  v.JFT? 

aP2(x)  _  ■  0Mi  • ‘Khm) * Khm) - (j3 *j3>) 

3j  f  \A2-  /-2  .2 

f-M  i  - Vi  +  j 

4.3.  Optical  and  Media  Errors 

In  the  simulation  of  realistic  optical  effects,  errors  arise  due  to  uncertainty  in  camera  parameters  such  as  focal 
length  or  magnification.  Additional  error  sources  in  MLI  simulation  and  error  analysis  models  comprise  inaccurate 
estimation  of  media  optical  parameters,  such  as  the  scattering  or  absorption  coefficient.  We  discuss  such  errors  in  the 
following  development. 

In  Equation  (VII),  if  the  lens  focal  length  or  magnification  is  erroneously  specified  or  measured,  then  taking  the 
derivative  of  x  e  X  with  respect  to  f  or  M  yields  the  following  optical  system  errors,  expressed  vectorially: 


and 


[2  ,  .2 

+  J 

f  M2i 


Lens  errors  and  distortions  are  not  the  only  optical  effects  that  can  degrade  the  accuracy  of  target  reconstruction. 
For  example,  media  effects  such  as  scattering  and  absorption  can  corrupt  the  received  image,  thereby  inducing  spatial 
and  greyscale  perturbations  in  the  focal  plane  when  media  optical  parameters  are  erroneously  prespecified  or  inaccu¬ 
rately  measured.  In  practice,  due  to  errors  resulting  from  current  in  situ  sensor  technology,  media  optical  parameters 
can  only  be  estimated.  Hence,  it  is  useful  to  consider  the  error  involved  in  modelling  optical  effects. 


Let  a  volume  scattering  function  p  :  (-n,  7t]  — >  [0,  1 J  generate  the  weights  of  a  template  s  :  R+  x  X  RX  that 
instantiates  a  linear,  single-scattering,  near-field  approximation  of  P  .  Denoting  the  terminus  of  a  scattering  process’ 
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optical  axis  as  y  e  X  and  the  terminus  of  a  ray  scattered  off-axis  as  x  e  X ,  a  scattering  template  s  is  defined 
in  terms  of  its  weights  as: 


s  (d)(x)  =  P($),  where  £  =  tan 


if  1  y-x 


with  d  e  R  and  x,  y  e  X  . 


Error  Analysis,  It  is  possible  that  the  scattering  length  d  can  be  incorrectly  estimated  from  the  corresponding  scattering 
coefficient  b,  since  the  mean  value  of  d  for  a  volume  scattering  function  (3  is  derived  as  d  =  1  /b  .  Error  analysis  of 
the  template  definition  in  Assumption  2.4.3  proceeds  as  follows.  Letting  v  =  ||  y  -  x  || ,  the  error  3sy(x)  in  a  given 

template  weight  due  to  a  perturbation  3b  in  the  scattering  coefficient  is  given  by 

3sy(x)  ^  D(p)  •  v1/2  *  tan  *(v1/2-b) 

=  777:  ■ 


where  D((3)  denotes  the  derivative  of  (3(i3) ,  at  $  corresponding  to  x  and  y,  per  Assumption  2.4.3.  Within  the  near 
field  of  p,  the  approximation  x  =  tan(jc)  holds,  which  yields  the  following  simplification: 

dsy(x)  D(P)  •  vb  D(P)  vb  _  D(P)  •  b 


3b  1  ,  ,2  .  -1  ,2.  -1  ,2  * 

1+bv  v(  v  +b  )  v  +  b 

This  indicates  that  the  error  due  to  scattering  varies  inversely  with  b  and  directly  with  v,  the  square  of  the  distance 
between  x  and  y.  In  practice,  this  means  that  the  intensity  error  in  the  scattering  template  is  smaller  for  turbid  media 
(larger  b),  which  directly  implies  shorter  scattering  distances. 

The  spatial  effects  of  scattering  are  similarly  described.  For  example,  the  scattering  angle  can  be  expressed  as 
=  tan  ~\Jv/ d)  =  tan-1(Vbv) ,  which  yields  the  result  d  =  Jbv  .  Hence,  the  displacement  perpendicular  to  the 
optical  axis  is  given  by  v  =  0  /b ,  which  exhibits  the  following  dependencies  upon  b  and  1}  : 


3v  _  f 

3b  vb  J 


,  3v  2$ 

and  =  T  ' 


Note  that  3v  increases  supralinearly  as  b  decreases  and  $  increases.  Hence,  narrow-angle  scattering  in  turbid  water 
would  yield  more  accurate  simulation  of  lateral  displacement  due  to  scattering  effects.  This  is  consonant  with  our  pre¬ 
ceding  assumption  of  the  near-field  approximation. 

Absorption  is  generally  characterized  by  the  coefficient  a,  which  yields  attenuation  a  over  a  propagation  distance 
d  that  is  given  by  a  =  e~ad .  , 

Error  Analysis.  Given  the  preceding  observation,  taking  the  derivatives  of  a  with  respect  to  a  and  d ,  we  obtain 


3a  ,  -a  d 
-  ~d  ■  e 
3a 


,  3  a  -ad 

and  tt—  =  -a  •  e 

ad 


In  practice,  this  means  that  the  error  involved  in  estimating  the  absorption  effect  decreases  with  increasing  a  and  d. 
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since  the  exponential  function  predominates. 

In  practice,  modelling  of  scattering  and  absorption  is  performed  by  linear  convolution  and  multiplication  opera- 
X 

tions.  For  example,  let  b  e  R  denote  an  undistorted  image  whose  pixels  have  been  formed  by  a  projective  process 
that  is  applied  concurrently  with  scattering  and  absorption  processes.  Given  template  s  defined  in  Equation  (IX)  and 

the  absorption  constant  a,  together  with  an  array  of  propagation  distances  d  e  (R+)  ,  single  scattering  and  absorption 
are  simulated  as  follows: 

c  =  (b  ■  e~ad)  ®  s(d)  . 

This  approximation  has  been  verified  as  physically  accurate  for  simulating  the  effects  of  near-field  forward  scattering 
in  moderately  turbid  seawater  and  atmospheres  [Sch96c],  and  is  hence  useful  for  the  purposes  of  this  preliminary 
study. 

4,4,  Camera  Errors 

Physical  imaging  sensors  generally  have  numerous  error  sources  that  directly  and  adversely  impact  the  recon¬ 
struction  of  digitally  imaged  targets.  Included  in  these  error  sources  are  the  following  primary  effects: 

1)  Intensifier  noise  produced  by  avalanche  phenomena  inside  the  microchannel  tubules  of  intensifier  tubes  ap¬ 
pears  to  provide  the  primary  contribution  to  image  intensifier  noise.  Often  described  in  terms  of  Gaussian 
statistics  for  convenience,  intensifier  noise  has  also  been  characterized  in  terms  of  Lorentzian  distributions 
[Sar92]; 

2)  Nonuniform  pixel  gain,  which  tends  to  decrease  toward  the  detector  periphery,  resulting  in  vignetting  of  the 
majority  of  non-zero  or  non-saturated  image  frames  output  by  a  digital  camera; 

3)  Pixel  errors  such  as  missing  pixels  or  (infrequently)  missing  rows  and  columns  of  pixels;  and 

4)  Preamplifier  noise,  which  tends  to  exhibit  Gaussian  or  Poisson  statistics  as  a  function  of  spatial  coordinate. 
Camera  errors  can  thus  be  concisely  described  by  a  noise  map  bn ,  gain  map  bg  and  pixel  defect  map  bp .  For 

example,  if  a  noiseless  (synthetic)  camera  image  a  is  a  real-valued  image  on  an  Mc  x  N£  -pixel  domain  X,  then 
bn,  kg’  bp  g  R  and,  assuming  additive  noise,  the  simulated  (noise-corrupted)  image  d  is  given  by: 

d  =  bp  •  bg  •  (a  +  bn)  , 

where  bp  and  bg  are  prespecified  and  can  correspond  to  a  given  type  of  detector.  In  contrast,  bn  is  specified  in  terms 

of  a  Gaussian  distribution  about  a  random  variable  r(x)  e  R,  x  e  X  ,  as 

-(r(x)-n)2/(2o)2 

Pr(r(x))  =  - — -  , 

flu  ■  a 

where  p  and  o  signify  the  mean  and  standard  deviation,  respectively,  or  by  a  Lorentzian  distribution 

Pr(Kx))  =  - -  . 

7t(r~(x)  +  or) 
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4.5.  Glint  Effects 

When  a  target  or  its  background  exhibits  specular  reflectance  (e.g.,  glint  from  a  vehicle  windshield  or  the  corru¬ 
gated  surface  of  a  wind-rippled  pond),  such  reflectance  effects  can  enhance  or  degrade  target  imagery.  For  example, 
glint  from  metallic  parts  on  a  vehicle  disguised  by  partially  opaque  camouflage  (e.g.,  netting)  can  provide  structured 
information  regarding  key  target  features.  Similarly,  glint  from  background  proximal  to  a  camouflaged  target  could 
provide  information  about  possible  target  presence  and  shape  (e.g.,  in  non-glint  areas). 

Alternatively,  glint  collocated  with  or  surrounding  small  targets  can  cause  saturation  proximal  to  target  pixels  in 
a  received  image.  This  instance  of  glint  has  the  usual  effect  of  reducing  signal-to-noise  ratio  in  the  neighborhood  of 
the  glint  pixels,  due  to  inter-channel  crosstalk.  Such  phenomena  tend  to  be  amplified  by  intensifier  tubes,  as  discussed 
in  Reference  [Sch96d]. 

In  the  computer  model  developed  for  this  study,  glint  is  simulated  as  follows: 

Step  1 .  Assume  that  there  exists  a  light  source  at  point  pL  =  (xL,  yL,  zL)  and  an  imaging  sensor  at 

Pi  =  (xj.yj.Zj). 

Step  2.  For  each  target  point  x  =  (x,,  yt,  z{)  e  X ,  compute  the  angles  8  and  f,  as 


Step  3- 
Step  4. 


5  =  tan 


7(x  L~xt)2  +  (yL-yt)2^ 

ZL~zt 


and  (})  =  tan 


-1 


V(xi-V2  +  (yj-yt)2 


Zj-zt 


Compute  glint  propagation  angle  as  £;  =  |5  —  <J)|  . 

Assuming  the  existence  of  a  radially  symmetric  angular  reflectance  (glint)  response  function 
y  :  (—71,  Ti  ]  — >  f  0,  1  ] ,  determine  gg  =  y(2;) ,  which  is  the  gain  due  to  glint  at  target  point  xe  X. 


Step  5 


Given  a  target  reflectance  map  re  [0,  1] 


X 


,  and  a  light  source  of  intensity  Is ,  compute  the  glint  image  as 


gs{(x,  g(x))  :  g(x)  =  IS  gg(x)  r(x),x€  X}  .  (X) 

Observe  that  several  simplifying  assumptions  have  been  implemented  in  the  preceding  algorithm,  as  follows: 

1)  A  spatially  uniform  reflectance  function  y  is  assumed,  which  does  not  occur  in  practice,  due  to  spatial  vari¬ 
ance  in  composition  of  scene  materials.  For  example,  the  specular  reflectance  of  a  vehicle’s  windshield  is 
more  sharply  peaked  about  2;  =  0  that  the  generally  Lambertian  reflectance  of  treetops  or  a  grassy  field  sam¬ 
pled  at  spatial  scales  ranging  from  1m  to  10m. 

2)  The  reflectance  angle  2;  is  not  computed  with  respect  to  a  non-horizontal  target  or  background  surface  normal 
at  a  given  target  point  x  e  X  .  Although  the  ensemble  average  (i.e.,  a  horizontal  target  or  background  surface) 
is  assumed,  the  majority  of  surface  normals  are  non-vertical,  especially  in  foliated  scene  neighborhoods. 

3)  A  point  light  source  is  assumed,  and  is  implemented  by  not  accounting  for  illuminant  divergence,  as  well  as 
camera  admittance  angle.  This  simplifying  assumption  reflects  the  emphasis  of  this  study  on  first-  and  sec¬ 
ond-order  effects,  namely,  specular  reflectance  in  the  case  of  glint  effects. 


Schmalz,  M.S.  —  Multi-Look  Imaging  —  AFOSR  Summer  Research  Extension  Program  Final  Report,  Dec.  1998 

10-27 


28 


We  next  discuss  the  rationale  for  each  of  the  foregoing  assumptions. 

Spatial  nonuniformity  of  the  surface  reflectance  angular  response,  while  realistic,  can  be  difficult  to  simulate  in 
an  optical  effects  model  not  specifically  associated  with,  or  compatible  with,  a  3-D  computer  graphics  or  rendering 
model  or  software  package.  In  particular,  a  3-D  model  customarily  specifies  target  and  background  objects  in  terms 
of  an  ensemble  of  spatially  disjoint  facets,  each  of  which  can  have  a  surface  normal  that  is  distinct  from  the  surface 
normals  of  spatially  adjacent  facets.  Although  we  have  shown  that  this  simplified  modelling  approach  facilitates  ac¬ 
curate  approximation  of  continuous  surfaces  by  finite  discrete  models,  the  computational  overhead  required  to  render 
such  views  can  be  prohibitively  large.  In  this  study,  we  emphasize  the  fast  modelling  of  different  MLI  scenarios,  for 
purposes  of  comparing  and  contrasting  target  visibility,  as  well  as  providing  large  databases  for  follow-on  error  anal¬ 
ysis.  The  usual  rendering  cost  was  not  acceptable  under  such  constraints,  and  we  chose  instead  the  simpler  (but  sta¬ 
tistically  accurate)  method  of  a  uniform  reflectance  response  function  referenced  to  the  ensemble  average  of  surface 
slopes.  In  future  versions  of  this  model,  we  plan  to  introduce  the  more  realistic  treatment  afforded  by  a  rendering  mod¬ 
el,  as  time  and  support  permits. 

The  assumption  of  a  point  light  source  is  acceptable  for  active-source  illuminant  models,  and  as  a  first  approxi¬ 
mation  to  insolation  (mcomi.ig  solar  radiation)  in  the  presence  of  a  clear  atmosphere.  Since  the  vast  majority  of  spec¬ 
ular  reflectance  effects  occur  in  the  presence  of  point  or  extended-point  sources,  the  point-source  assumption  is 
reasonable  for  this  first-order  model.  We  plan  to  extend  this  to  extended-point  and  diffuse  sources,  as  time  permits. 
4,6.  Effect  of  Cover  Aperture  Size  and  Spacing 

The  spatial  extent,  interstitial  spacing,  and  transmittance  or  cover  objects  can  vary  widely  in  practice.  In  the  initial 
part  of  this  study,  we  examined  the  effects  of  cover  object  size  and  spacing  (e.g.,  tree  diameter  and  distance  between 
centers  of  tree  trunks),  attempting  to  determine  if  there  was  a  correlation  between  cover  object  size  and  separation  and 
target  visibility.  What  was  discovered  in  later  parts  of  this  study  was  that  the  cover  aperture  size  and  spacing  was  the 
chief  determinant  of  target  visibility  in  a  given  sensing  scenario.  In  particular,  determining  the  overlap  effect  of  suc¬ 
cessive  views  or  looks  as  a  function  of  aperture  parameters  proved  to  be  a  more  effective  analytical  perspective. 

For  example,  consider  the  circular  aperture  shown  in  Figure  7a.  Given  a  linear  track  with  views  closely  (widely) 
separated  in  space,  the  composite  aperture  effects  shown  in  Figure  7b  (7c)  would  result,  all  other  parameters  being  held 
constant.  Qualitative  observation  notes  that  wide  spacing  of  small  apertures  tends  to  offer  little  or  no  advantage  in 
terms  of  obtaining  multiple  views  per  given  target  pixel,  but  nevertheless  provides  greater  total  coverage  in  the  target 
plane  across  all  views  than  w  mid  be  obtained  by  any  single  view.  Given  the  geometry  of  Figure  1,  it  is  possible  to 
estimate  the  interframe  distance  along  a  sensor  track  required  to  achieve  minimal  overlap,  as  follows. 

Let  circular  apertures  of  diameter  gc  meters  be  located  at  an  altitude  aQ  meters  above  the  target  plane,  over  which 
a  sensor  flies  at  altitude  a&  meters  along  a  linear  or  quasi-linear  track  that  crosses  the  centers  of  the  apertures,  as  shown 
in  Figure  8a.  Here,  the  assumption  of  bisection  is  made  to  simplify  the  following  analysis,  and  states  the  ensemble 
average.  Assuming  nadir  viewing  to  minimize  the  keystoning  effect,  observe  (from  Figures  1  and  8b)  that  the  cover 
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aperture  width  projected  to  a  flat  target  plane  is  given  by  the  following  expression  for  the  mean: 


(XI) 


It  is  readily  determined  by  inspection  of  Figure  8b  that  the  mean  interframe  distance  along  a  sensor  track  that  the  mean 
interframe  distance  along  the  sensor  track  required  to  achieve  frame  overlap  (and,  hence,  multiple  views  of  at  least  n- 
1  target  pixels  given  n  sensor  positions)  is  given  by 


dif^gt  • 

The  interframe  overlap  effect  is  closely  related  to  sensor  speed  s  and  frame  rate  F,  as  shown  in  the  following  equation: 


The  following  example  is  illustrative. 


Figure  7.  Effect  of  aperture  separation:  (a)  simulated  cover,  (b)  widely  spaced  apertures  projected 
to  the  target  plane,  (c)  closely  spaced  apertures  projected  to  the  target  plane. 


X 

Sensor 

Track 


(a) 


Eigur.e.8.  Geometry  of  multi-view  overlap  in  successive  frames:  (a)  sensor  track  crosses  centers  of  circular  apertures, 
and  (b)  sensing  geometry  for  determining  interframe  distance  dif  to  achieve  interframe  overlap. 

Example.  Let  as  =  1km,  ac  =  20m,  and  gc  =  10m.  Then,  djf  =  1 000(  1 0)/980  =  10.2  meters.  Assuming  a  sensor 
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velocity  across  the  midpoints  of  adjacent  apertures  having  speed  component  of  s  =  200  mph  =  94.1  meters/second,  the 
frame  rate  required  to  achieve  this  interframe  distance  would  be  at  least 

F  =  |"s/dif]  =  T94.1  m/s/ 10.2m "1  =  10  frames/sec  . 

Error  Analysis.  Taking  the  derivative  of  Equation  (XI)  with  respect  to  ac  and  as  yields: 


dg, 


( as~ac ) 


and 


3  St 


(as~ac) 


which  means  that  as  as  approaches  aQ ,  the  preceding  error  measures  will  become  unstable.  In  practice,  this  indicates 

that  interframe  overlap  can  increase  drastically  as  the  sensor  platform  approaches  the  cover  from  above  (e.g.,  flies  just 
above  the  treetops).  However,  there  also  exists  the  practical  problem  that  objects  in  the  target  plane  are  occluded  from 
oblique  viewing  by  the  sensor  by  structures  such  as  tree  trunks,  brush,  etc.  This  effect  is  discussed  in  the  following 
paragraphs,  and  constitutes  a  significant  complication  in  the  simulation  of  low-altitude  MLI. 

An  important  consideration  in  MLI  with  linear  tracks  is  the  effect  of  target  occlusion  by  three-dimensional  cover 
objects  such  as  the  vertical  or  nearly-vertical  treetops  of  coniferous  forests  (as  illustrated  schematically  in  Figure  9a) 
as  opposed  to  the  spherical  or  pear-shaped  cross-sectional  profiles  of  deciduous  trees  (Figure  9b).  We  have  found  that 
cover  objects  with  flatter  vertical  cross-sectional  profiles  (e.g.,  reduced  height-to-diameter  ratio  rhd)  tend  to  hide 
ground  targets  less  well  than  generally  conical  profiles  of  coniferous  trees  (larger  rhd),  other  parameters  being  equal. 

In  Equation  (XI),  the  variable  gc  is  assumed  to  be  measured  along-track.  It  would  appear  to  a  naive  observer 

that  interframe  overlap  could  be  maximized  when  the  sensor  track  follows  the  projection  of  the  longest  cover  seg¬ 
ments.  However,  this  is  not  necessarily  true,  due  to  3-D  effects  incurred  by  the  presence  of  projecting  cover  objects 
that  are  constituent  to  the  cross-sectional  geometry  of  various  cover  objects,  for  example,  as  shown  in  Figure  9c,  it 
may  be  advantageous  for  sufficiently  agile  sensors  to  descend  into  the  region  of  target  visibility  delimited  by  the  pro¬ 
jection  of  cover  boundaries  upward  from  target  objects  to  the  sensor  track.  In  contrast,  our  development  in  this  report 
has  assumed  projection  of  cover  segments  downward  to  the  target  plane  from  the  reference  location  of  the  imaging 
sensor’s  nodal  point,  in  order  to  determine  target  visibility.  However,  when  determining  optimal  sensor  track,  reverse 
projection  is  useful,  because  it  provides  information  about  regions  of  target  visibility  into  which  an  airborne  sensor 
could  fly,  to  obtain  optimal  target  imagery  (i.e.,  with  significantly  reduced  probability  of  occlusion).  This  concept  is 
further  discussed  in  Sections  3.1,  5.2,  and  6.3,  after  we  present  simulation  methodology  and  examples. 

4.7.  Target  Visibility  and  Error  Simulation  Model 

As  shown  in  Figure  10,  the  concept  of  MLI  simulation  in  the  context  of  this  study  is  to  combine  multiple  views 
of  an  object  to  determine  (a)  target  visibility  and  (b)  error  of  estimation  associated  with  the  visibility  estimate.  The 
latter  measure  is  especially  important  when  determining  the  potential  accuracy  of  target  reconstruction  methods,  such 
as  presented  in  Section  5. 
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Figure  10,  MLI  simulation  concept  overview,  where  the  merge  step  denotes  a  target  visibility  estimation  algorithm. 

We  have  constructed  a  simulation  model  in  computer  software  that  estimates  a  target  visibility  map  as  well  as  an 
error  budget  for  MLI  given  user-specified  sensor  track,  cover  configuration  and  altitude,  atmospheric  optical  parame¬ 
ters,  camera  noise  and  pixel  defect  parameters,  and  glint  effects.  This  model  has  the  structure  shown  in  Figure  11. 
observe  the  parallel  functionality  for  image  simulation  and  error  estimation,  which  facilitates  the  verification  of  target 
visibility  information  at  a  given  point  x  e  X  of  the  target  plane  by  consulting  the  error  estimation  map  at  x.  Addition¬ 
ally,  the  modular  structure  of  this  model  facilitates  maintenance  and  expansion  to  meet  future  research  needs. 


Error  Estimation  Partition  Image  Simulation  Partition 


Figure  1 1 .  MLI  simulation  model  architecture,  showing  concurrent  error  estimation  and  image  simulation. 
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Figure  12.  Schematic  diagram  of  projection  process  that  forms  target  visibility  map. 

Target  visibility  is  simulated  by  projecting  each  sensor  pixel  boundary  at  each  sensor  position  pp  i  =  0..n-l ,  to  the 
target  plane.  The  accumulation  of  cover  transmittance  adjusted  for  atmospheric  effects  comprises  the  basic  target  vis¬ 
ibility  map.  This  process  is  illustrated  schematically  in  Figure  12.  Error  maps  are  produced  that  account  for  camera 
and  glint  effects  within  the  accuracy  of  approximations  discussed  in  Sections  4.4  through  4.6.  Such  error  maps  can  be 
combined  with  the  basic  target  visibility  map,  for  analytical  purposes,  as  discussed  in  Section  5. 

Algorithm.  The  basic  projection  algorithm  proceeds  as  follows.  Given  an  MxN-pixel  cover  image  cg  [0,  1  ]Y  ,  where 
M  and  N  are  prespecified;  a  sensor  track  comprised  of  a  sequence  of  three-dimensional  points  P  = 
{P0>  P2>  Pk’  •’  Pn-  i  }*  where  pk  =  (xk,  yk,  zk) ;  resolution  intervals  at  the  cover  or  target  planes  Axc,  Ayc  or 

Axt,  Ayt ,  with  cover  and  altitudes  denoted  by  zc,  zt  €  R+ ,  perform  the  following  steps: 


Step  1. 

Initialize  b,  c  :=  0,  the  zero-valued  constant  image. 

Step  2. 

for  k  =  0  to  n-1,  do: 

(2a) 

if  Axt,  Ay}  are  unspecified,  then  Axt  =  Axc  * : 

zs/(zs-zc)  and  Ayt  =  Ayc  - zs/(zs-zc) 

(2b) 

for  i  =  1  to  M  do: 

I 

r  x  -x_ i 

(2c) 

X 

o 

II 

> 

X 

n 

II 

(z,  zt)' 

L  zs  -  ZcJ 

+  xs  ;  i(  =  nint(x{/Ax{) 

(2d) 

for  j  =  1  to  N  do: 
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r  yc-  y.i 

(2e)  yc  =  Ayc-j;  yt  =  (zs-z,)-  — — —  +ys;  jt  =  n/nr(yt/Ayt) 

L  zs  zcJ 

(2f)  a(i,  j)  =  (xt,yt,zt)  ;  ds( i,  j)  =  fl  pk-a(i,  j)  || 

(2g)  if  i  >  1  and  j  >  1,  then 

(2g.  1 )  Construct  a  polygon  (rectangle,  triangle,  or  trapezoid)  from  a(i  j)  and  the  adjacent  pix¬ 

el  vertices  a(i~l,j),  a(i,j-l),  and  a(i-lj-l).  Denote  the  points  in  X  contained  in  this 
polygon  by  N(i,j). 

(2g.2)  Let  the  accumulated  transmittance  image  d  be  augmented  as 

dL„.  ■  \  =  d I ....  .  +  c(i,  j)  . 

W(ipj)  W(i»j) 

(2g.3)  Let  the  accumulator  image  for  the  number  of  views  per  target  pixel  be  augmented  as 

b I ....  ..  =  b  I ....  ..  +  1  . 
w(»,j)  wftj) 

(2h)  endif ;  endfor  ;  endfor  ;  endfor 

Step  3.  Calculate  the  mean  transmittance  per  target  pixel  as  f  =  d  /  b 

Step  4.  Given  the  scattering  template  s  defined  in  Equation  (IX)  as  well  as  scattering  and  absorption  coeffi¬ 
cients  b  and  a,  perturb  f  to  portray  the  effects  of  scattering  and  absorption,  as  follows: 

(4a)  Compute  the  order  of  scattering  ks  =  ds(i,  j)  •  b . 

- -  k,  times - 1 

-a d.  I  s  I 

(4b)  Let  the  simulated  result  be  given  by  hk  =  (f  e  )  ©  s(ds)  ©  s(ds)  0  ...  ©  s (ds) 

Step  5.  Combine  the  individual  maps  h^,  for  k  =  0..n-l,  via  pointwise  summation  over  X,  that  is 


h  =  |  (x,  h  (x))  :  h  (x)  =  hk(x),  xgX  . 

I-  k  =  0  -I 

X 

Step  6.  Calculate  the  camera  noise  as  shown  in  Equation  (X),  to  produce  a  camera  noise  map  nk  e  R  for  each 

of  k  =  0..n-l  views.  Combine  all  nk  by  summation,  as  in  Step  5,  to  yield  a  composite  noise  map  nT  that 
portrays  additive  accumulation  of  noise  between  frames. 

Step  7.  Calculate  the  glint  effects  as  shown  in  Section  4.5,  to  produce  a  glint  effects  map  mk  €  R  for  each 

view.  Combine  the  noise  maps  by  summation,  as  in  Step  5,  to  yield  a  composite  glint  map  in  ,  which 
portrays  additive  accumulation  of  glint. 

The  function  of  each  algorithm  step  is  explained  as  follows. 

All  data  stated  as  givens  are  read  from  a  control  file,  or  from  ancillary  files  referenced  by  filename  in  the  control 
file.  This  allows  various  cover  configurations,  noise  and  glint  parameter  sets,  and  sensing  geometries  to  be  specified 
in  a  flexible  way.  In  Step  1,  the  counter  image  b  and  the  transmittance  accumulator  image  d  are  initialized  to  zero, 
which  is  appropriate  for  accumulators.  Step  2  begins  with  the  k-loop,  which  directs  the  simulation  by  stepping  through 
the  multiple  views  possible  from  the  sensor  positions  in  P.  Step  2a  computes  the  target-plane  instantaneous  field-of- 
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view  (IFOV)  in  the  *-  and  y- axis  directions,  if  such  components  are  not  specified  in  the  control  file.  Step  2b  iterates 
through  the  rows  of  pixels  in  the  cover  image,  and  Step  2c  contains  computations  of  the  cover  pixel  coordinate  xc, 
target  pixel  coordinate  xt,  and  target  pixel  index  it.  Here,  the  nint(x)  function  computes  the  integer  nearest  the  real 
argument*.  Steps  2d  and  2e  are  symmetric  to  Steps  2b  and  2c,  but  apply  to  column  coordinates.  In  Step  2f,  the  target- 
plane  coordinates  are  stored  in  a,  and  this  information  is  transformed  into  a  propagation  distance  stored  in  ds,  which 
is  computed  between  the  sensor  position  pk  and  the  target-plane  coordinate  (xt,  yt,zt). 

If  processing  is  not  occurring  within  the  first  row  or  column  of  c,  then  it  is  possible  to  construct  the  projection  of 
the  current  cover  pixel  to  the  target  plane.  This  is  done  by  computing  the  target  plane  loci  within  the  polygonal  neigh¬ 
borhood  N  whose  vertices  are  given  by  a(i,j),  a(i-l,j),  a(i,j-l),  and  a(i-l ,j-l).  In  cases  where  three  vertices  are  colinear, 
N  has  triangular  shape,  otherwise  N  is  rectangular  or  trapezoidal.  The  determination  of  N  usually  represents  the  most 
computationally  costly  step  of  the  projection  algorithm,  since  line  intersections  must  be  computed,  and  a  version  of 
Bresenham’s  algorithm  for  computing  the  points  on  a  straight  line  of  arbitrary  slope  must  be  employed  [Kuz95].  Once 
the  points  within  the  aforementioned  polygon  are  collected  in  N,  the  updates  to  b  and  d  can  be  efficiently  computed. 
Step  2h  is  merely  the  termination  of  nested  if  and  for  loops. 

The  physical  justification  for  accumulation  of  transmittance  in  d  (Step  2g.2)  may  not  be  immediately  apparent, 
but  can  readily  be  seen  if  one  considers  the  effect  of  a  light  source  of  constant  intensity  positioned  at  each  of  the  points 
of  P,  which  source  shines  through  the  cover  apertures.  Clearly,  the  accumulated  irradiance  at  any  point  in  the  target 
plane  would  be  directly  proportional  to  the  accumulated  cover  transmittance  projected  to  that  point.  Similarly,  the  av¬ 
eraging  of  accumulated  transmittance  in  Step  3  may  not  be  immediately  obvious,  but  can  be  readily  understood  by 
noting  that  b  contains  the  number  of  ray  bundle  projections  through  the  pixels  of  the  cover  plane  to  a  given  point  in 
the  target  plane.  Since  range( b)  =  [0,n],  pointwise  division  of  d  by  b  yields  the  average  cover  transmittance  at  each 
point  of  X  for  the  n  simulated  views.  A  task  for  future  research  would  emphasize  determination  of  whether  or  not  the 
mean  is  the  most  effective  measure  in  this  situation  ~  for  example,  should  the  median  or  some  statistical  characteriza¬ 
tion  (e.g.,  m-tile  boundaries)  be  respectively  employed  to  characterize  MLI  in  the  presence  of  small  or  large  n. 

The  simulation  of  multiple  scattering  by  repeated  application  of  the  linear  model  described  in  Section  4.3  has  been 
a  feature  of  our  aqueous  scattering  models  for  the  past  eight  years,  and  has  been  verified  in  practice  as  being  sufficient¬ 
ly  accurate  within  two  to  three  attenuation  lengths  that  estimation  error  is  within  the  greyscale  quantization  interval 
(i.e.,  0.39  percent  of  peak  intensity)  of  an  eight-bit  digital  image  [Sch96c].  Since  three  orders  of  multiple  scattering 
rarely  occur  in  airborne  imaging  through  clear  or  somewhat  hazy  atmospheres  (where  the  scattering  coefficient  b  is 
small  in  relationship  to  its  value  in  aqueous  media),  the  accuracy  of  this  approximation  for  near-field  forward  scatter¬ 
ing  should  not  be  a  primary  concern  in  the  analysis  of  simulation  error. 

The  additive  combination  of  average  transmittance  maps  is  justified  in  this  preliminary  model  by  observing  that 
the  component  maps  hk  are  computed  via  the  arithmetic  mean,  and  hence  have  addition  as  a  component  generative 
operation.  The  additive  combination  of  noise  or  glint  component  maps  to  yield  a  composite  map  is  consonant  with  the 
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observation  that  noise  tends  to  accumulate  additively  in  imaging  systems,  particularly  in  intensified  cameras 
[Sar92,Sch96e],  and  the  preceding  observation  concerning  additive  accumulation  of  irradiance  (of  which  glint  is  an 
instance). 

We  next  present  several  simple  examples  of  the  analysis  of  simulation  model  output. 

5.  Visibility  and  Error  Simulation  and  Analysis 

In  this  section,  we  summarize  results  obtained  from  numerous  model  simulation  runs  and  analysis  of  MLI-based 
target  reconstruction  algorithms.  Section  5.1  discusses  an  example  loitering  scenario,  while  imaging  along  a  linear 
track  is  analyzed  in  Section  5.2.  Such  discussion  emphasizes  different  modes  in  which  the  simulation  model  can  be 
employed  to  focus  on  the  effect  of  specific  physical  phenomena  and  associated  constraints.  In  Sections  5.3 , 5.4,  and 
5.5,  we  respectively  present  error  analyses  of  target  reconstruction  based  on  cepstral  coregistration,  relative  geometry, 
and  tomography  at  unequally-spaced  sensor  positions. 

5.1.  Example  Simulation:  Loitering  Scenario 

Let  a  sensor  comprised  of  an  imaging  device  having  unitary  magnification,  infinite  depth-of-field  (idealized  pin¬ 
hole  camera),  and  Gaussian  noise  of  mean  0.1  and  standard  deviation  0.02  over  a  greyscale  interval  [0,1]  be  positioned 
according  to  the  curvilinear  track  listed  in  Table  1,  which  is  a  segment  of  a  loitering  track.  Simulated  sensor  altitude 

is  300m,  cover  altitude  is  20m,  and  atmospheric  absorption  and  scattering  coefficients  were  set  at  a  =  0.003  m'1  and  b 
=  0.002  m_1.The  cover  image  and  the  composite  visibility  image  are  shown  in  Figure  13.  Note  how  the  overlay  of  each 
projected  cover  aperture  accumulates  in  the  composite  image,  where  bright  (dark)  pixels  correspond  to  target  regions 
where  many  (few)  views  are  available.  The  data  of  Table  2  are  graphed  in  Figure  13c. 

Table  1.  Example  loitering  scenario:  Sensor  track  points. 


Track  Point 

xs,  meters 

ys,  meters 

zs,  meters 

Po 

228 

0 

300 

Pi 

150 

50 

240 

P2 

128 

100 

200 

P3 

114 

150 

270 

P4 

177 

200 

290 

Ps 

310 

250 

300 

The  probability  of  a  given  number  of  views  for  this  example  scenario  is  expressed  in  Table  2.  Note  that  the  large 
probability  of  no  views  (Pr(0)  =  0.956)  is  a  function  of  the  size  and  resolution  of  X.  For  example,  if  the  brightest  region 
of  Figure  13b,  corresponding  to  a  total  of  six  views  per  pixel,  contains  1,000  pixels  and  X  is  1024x1024  pixels  in  size, 
then  Pr(6)  =  0.001 .  However,  if  X  is  increased  to  2048x2048  pixels,  then  Pr(6)  =  0.00025,  all  other  parameters  being 
held  constant.  Clearly,  the  inclusion  of  Pr(0)  biases  the  remaining  probabilities  and  is  thus  eliminated  in  Figure  13c, 
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by  normalizing  each  probability  Pr(i),  i  =  l..n,  to  the  maximum  of  Pr(i).  Alternatively,  one  could  normalize  the  prob¬ 
abilities  to  the  sum 

n 

S  =  X  Pr(j)  =  l-Pr(O)  • 

j=  l 

Table  2.  Example  loitering  scenario:  Probability  of  target  visibility  given  a  number  of  views  n  =  6. 


Number  of  Projections  per  Pixel,  Nv 

Probability 

0 

0.956 

1 

0.009 

2 

0.012 

3 

0.005 

4 

0.009 

5 

0.006 

6 

0.003 

Figure  13.  Loitering  track  scenario:  (a)  cover  mask  (256x256  pixels),  and  (b)  projections  to  target 
plane  corresponding  to  the  statistical  data  of  Table  2,  and  (c)  graph  of  probability  of  projection  Nv  =  i 
where  i  =  1  ..6,  normalized  for  maximum  non-zero  projections,  versus  number  of  projections  Nv 


We  recommend  this  normalization  technique  for  future  analysis,  since  it  eliminates  bias  within  the  error  of  ±1  pixel 
incurred  by  sampling  over  the  IXI-pixel  target  domain. 


The  variation  in  the  extent  of  a  given  view  with  altitude  is  readily  ascertained  from  inspection  of  Figure  1 .  Ob¬ 
serve  that,  at  infinite  altitude  and  nadir  viewing,  the  cover  apertures  would  be  projected  directly  downward  to  the  target 
plane.  Although  this  case  does  not  occur  in  practice,  it  serves  to  illustrate  one  extreme  of  sensor  altitude,  at  which 
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Pr(Nv  =  1)  tends  toward  unity.  Tending  toward  the  other  extreme,  as  sensor  altitude  approaches  cover  altitude,  the 
projection  of  each  cover  aperture  to  the  target  plane  becomes  larger,  and  such  projections  tend  to  overlap  increasingly. 
This  effect  increases  the  number  of  views  for  which  Nv  >  1,  and  hence,  increases  Pr(Nv)  for  Nv  =  2..n. 

The  increase  in  number  of  potential  target  views  per  pixel  with  decreasing  altitude  is  shown  in  Figure  14b,  for  a 
cover  mask  similar  to  that  in  Figure  13,  but  with  multiple  apertures  and  partially  overlapping  circular  cover  segments. 
Cover  altitude  is  fixed  at  20m  for  all  sensor  altitudes,  and  the  target  plane  has  four  times  the  resolution  of  the  cover 
plane,  to  accommodate  the  expanding  projections  of  cover  apertures  as  sensor  altitude  is  decreased  (e.g.,  if  the  cover 
mask  is  256x256  pixels,  then  the  target  plane  has  1024x1024  pixels).  Because  the  cover  apertures  overlap  insufficient¬ 
ly,  Pr(n)  is  vanishingly  small. 


10  100  1000  10000  10  100  1000  10000 


as,  meters  as<  meters 

(a)  (b) 

Figure  14.  Variation  of  Pr(Nv)  with  sensor  altitude  as,  for  Nv  =  n/3  and  Nv  =  2n/3,  where  (a)  n  =  6  and  (b)  n  =  12. 

Note  that  Figure  14  also  portrays  the  effect  of  number  of  views.  For  example,  in  Figure  14b,  where  n  =  12,  the 
probabilities  for  Nv  =  n/3  =  4  and  Nv  =  2n/3  =  8  are  approximately  half  of  those  in  Figure  14a  where  Nv  =  2  and  Nv  = 
4.  This  scaling  of  Nv  and  Pr(Nv)  with  respect  to  n  appears  to  occur  throughout  our  simulation  runs,  albeit  with  more 
discrepancies  between  probabilities  associated  with  specific  instances  of  Nv  as  a  function  of  n.  Hence,  we  are  currently 
investigating  ways  to  scale  Pr(Nv)  with  respect  to  n.  For  example,  we  have  found  that  it  is  useful  to  measure  the  dif¬ 
ference  between  probabilities  obtained  during  simulation  runs  having  small  samples  (e.g.,  small  n),  with  the  following 
measure  of  scaled  probability: 

Mpn  =  n  •  Pr(/)  ,  where  Ny  =  I  n  varies  from  one  to  n. 

Note  that  Mpn  can  be  plotted  against  n  to  determine  when  the  number  of  views  is  sufficient  to  support  accurate  prob¬ 
ability  estimation.  For  example.  Figure  15  illustrates  the  difference  in  Mpn  between  successive  settings  of  Nv,  based 
on  the  simulation  configuration  from  which  Figure  13  was  generated.  Note  that  this  difference  measure,  denoted  by 
AMpn,  stabilizes  in  this  case  after  approximately  ten  views.  We  are  currently  investigating  conditions  for  the  stability 
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of  AMpn  as  a  function  of  computational  precision  in  other  simulation  scenarios. 

However,  the  graph  of  Figure  14  can  be  misleading,  since  it  foes  not  account  for  the  ability  to  see  obliquely  under 
trees  or  overhanging  cover.  As  shown  in  Figure  8b,  it  is  naively  assumed  that  if  sensor  altitude  approaches  cover  al¬ 
titude,  then  a  downlooking  sensor  can  view  objects  much  farther  away  from  a  cover  aperture  intersected  by  the  sensor’s 
optical  axis.  Unfortunately,  this  is  not  the  case  in  practice  (as  noted  previously)  because  tree  trunks,  brush,  and  other 
types  of  physical  clutter  interfere  with  the  sensor’s  oblique  view.  This  phenomena  increasingly  becomes  a  source  of 
occlusion  as  the  look  angle  for  a  given  pixel  in  the  sensor’s  focal  plane  increases.  This  leads  us  to  consider  a  simple 
modification  to  the  model  of  Section  4.7,  by  which  one  can  simulate  the  effect  of  three-dimensional  cover. 

Algorithm  Enhancement.  We  have  found  that  simulation  of  cover  as  a  stack  of  L  2-D  cover  masks  is  sufficiently  ef¬ 
fective  for  the  purposes  of  this  model,  provided  that  L  >  5  is  employed.  The  constraint  on  L  becomes  evident  when 
one  considers  that  a  tree  with  spherical  or  ellipsoidal  shape  will  have  a  bottom,  equator,  and  top,  with  one  slice  required 
between  the  bottom  and  eqtaior,  and  one  slice  between  the  equator  and  top,  to  portray  the  tree’s  cross  section  as  a 
polygon  approximating  a  round  object,  as  shown  in  Figure  16a-c. 


Figure  15.  Graph  of  AMpn  as  a  function  of  n  ranging  from  one  to  15  views,  based  on  the 
simulation  configuration  employed  in  the  generation  of  Figure  13. 


Top 


Km 

1™“™™“"““  Equator 


Bottom 

Trunk 


(a) 


(b) 


(c) 


Figure  16.  Approximation  of  3-D  arboreal  cover  cross-section  with  (a)  three,  (b)  five,  and  (c)  seven  2-D  masks. 
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The  enhancement  to  the  simulation  algorithm  required  to  compute  the  effect  of  3-D  cover  is  quite  simple,  and  is  based 
on  the  observation  that  any  downward  projection  through  any  3-D  cover  object  represented  as  a  stack  of2-D  masks 
must  penetrate  all  layers  of  cover  if  the  projection  intersects  the  target  plane.  In  practice,  this  means  that,  given  m 

y 

discrete  two-dimensional  cover  masks  e  [0,  1]  ,  where  the  stack  index  X  =  L.tn,  if  a  ray  traced  through  the  m 
masks  intersects  the  A,-th  mask  at  (i^,  j^)  e  Y ,  then  the  transmittance  t  associated  with  that  ray  is  given  by 

m 

t=  (xn) 

X  =  1 

which  follows  from  optics  theory. 

Hence,  instead  of  tracing  a  ray  from  a  sensor  position  p  to  a  point  in  the  target  plane,  we  perform  the  ray  tracing 
operation  L  times,  to  establish  the  set  of  coordinates 


m 

c( ij)  =  u  {(ivjx)},aj)€  y  . 

x  =  \ 

Then  one  performs  the  multiplication  shown  in  Equation  (XII)  to  obtain  the  transmittance  associated  with  the  target 
pixel  d(xt,yt),  whose  point  (xt,yt)  =  a(i,j)  is  colinear  with  p  and  the  points  in  C(i,j),  within  the  limits  of  colinearity  of 
digital  lines,  given  the  spatial  quantization  error  inherent  in  X  and  Y. 

It  is  of  further  interest  to  note  that  the  specification  of  a  cover  mask  over  the  range  space  [0,1]  permits  the  mod¬ 
elling  of  translucent  cover  (e.g.,  fog  or  haze),  as  well  as  the  statistical  characterization  of  cover  transparency  due  to 
partial  visibility  within  the  subtense  of  one  pixel  of  the  cover  mask  (at  IFOV  (Axc,  Ayc)  physical  units). 

Example.  Coniferous  tree  branches  that  are  opaque  but  occupy  only  k  percent  of  a  cover  pixel’s  subtense  at  the  cover 
plane  could  be  characterized  by  a  transmittance  of  kl  100  at  that  pixel  location. 

5.2.  Analysis  of  Cepstral-based  Coregistration 

Yeshurun  and  Schwartz  [Yes89]  describe  a  method  of  cepstral-based  coregistration  that  we  have  employed  in  the 
experiments  described  in  Section  6.1.  In  particular,  given  a  stereo  pair  of  MxN-pixel  images,  the  stereo  image  formed 
as  c  =  (a,b)  is  transformed  via  application  of  the  cepstrum  to  yield  the  disparity  image  d,  as  follows: 

d  =  J(log[|j(a)  +  l|])  , 

where  ^denotes  the  Fourier  transformation.  The  difference  between  the  coordinates  of  the  origin  and  largest  peak  of 
d  yields  the  stereo  disparity.  This  method  is  relatively  impervious  to  additive  noise,  as  shown  in  Figure  17a,  which 
sensitivity  analysis  results  incurred  in  correlating  two  instances  of  Figure  13b,  where  one  instance  is  corrupted  with 
Gaussian  noise  of  mean  0.1  on  a  scale  of  [0,1]  and  standard  deviation  0.02.  Figure  17b  (17c)  illustrates  correlation 
error  incurred  by  rotating  (scaling)  the  target  over  a  range  of  [-15,15]  degrees  (scale  factor  range  of  [0.7, 1.3]).  Note 
that  the  results  of  Figure  17c  for  scale  factor  values  less  than  0.8  or  greater  than  1.2  are  not  shown,  due  to  the  large 
errors  encountered  at  the  extrema  of  the  scaling  range. 
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Figure  17.  Correlation  error  for  Yeshurun  and  Schwartz’  cepstral  coregistration  of  the  pattern  represented  by  Figure 

13b  under  variable  (a)  noise,  (b)  rotation  angle,  and  (c)  scale  factor,  all  other  parameters  being  held  constant. 

The  preceding  analysis  is  presented  in  lieu  of  a  detailed  theoretical  treatment,  which  is  given  in  [Yes89]  and 
[Smi96].  A  key  problem  associated  with  analysis  of  the  cepstral-based  coregistration  method  is  the  analysis  of  error 
in  the  Fourier  transform,  whose  implementations  (and  associated  computational  error)  vary  widely.  This  variation  in 
computational  error  is  also  dependent  on  the  configuration  of  the  patterns  being  correlated  and  the  size  of  the  sample 
block.  For  example,  in  Figure  17,  we  employed  a  1024x1 024-pixel  image,  and  varied  the  sample  block  size  from 
16x16  pixels  to  64x64  pixels,  choosing  a  blocksize  of  32  pixels  square  as  optimal.  Slightly  different  results  were  ob¬ 
tained  with  different  sized  blocks,  as  shown  in  Figure  17a.  Note  that  the  64x64-pixel  block  contains  more  information 
(i.e.,  higher  spatial  frequencies)  than  the  smaller  blocks,  and  thus  produces  less  correlation  error  in  the  presence  of 
noise.  However,  the  computational  cost  associated  with  the  64x64-pixel  block  was  insufficient  to  allow  fast  imple¬ 
mentation  and  testing  for  the  required  number  of  simulation  runs. 

In  Figures  17b  and  17c,  it  is  shown  that  the  correlation  error  response  is  asymmetric  with  respect  to  the  indepen¬ 
dent  variable.  This  is  due  primarily  to  the  asymmetry  of  the  test  pattern,  and  to  local  periodicity  within  the  pattern 
(e.g.,  fine  structure  introduced  by  simulation  model  artifacts,  since  removed).  In  practice,  such  response  asymmetry 
would  result  when  correlating  realistic  targets  in  one  view  with  the  same  targets  in  another  view.  This  effect  was  ob¬ 
served  when  correlating  simulated  images  of  submerged  targets,  as  discussed  in  Section  6.1. 

Future  work  in  the  analysis  of  cepstral-based  correlation  involves  (a)  analyzing  error  propagation  in  different  im¬ 
plementations  of  the  Fourier  transform,  (b)  testing  Yeshurun  and  Schwartz’  algorithm  using  the  aforementioned  Fou¬ 
rier  transform  implementations,  as  well  as  (c)  implementing  and  testing  Smith  and  Nandhakumar’s  enhancement 
[Smi96]  of  Yeshurun  and  Schwartz’  algorithm  using  the  several  different  z-transform  implementations.  We  expect 
that  this  would  provide  a  more  extensive  database  of  test  results  against  which  to  estimate  performance  of  cepstral- 
based  coregistration  in  practice. 

5.3.  Analysis  of  Relative  Geometric  Reconstruction  Method 

The  change  in  the  2-D  projection  of  a  moving  3-D  object  or  camera  yields  important  information  for  3-D  object 
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Abstract 

The  command  and  control  (C2)  of  an  air  campaign  involves  different  processes:  planning,  scheduling,  execution, 
monitoring,  assessment,  and  re-planning.  The  monitoring  process  updates  the  situation  of  the  air  campaign  and 
detects  changes  in  the  campaign  by  processing  intelligent  information.  The  assessment  process  assesses  those 
changes  by  associating  the  current  situation  with  the  objective  of  the  campaign.  Together,  these  two  processes 
provide  appropriate  feedback  to  operation  commander  for  making  decisions  regarding  to  the  fate  of  the  air 
campaign.  This  project  proposes  methods  to  assess  a  campaign  by  fusing  intelligent  information.  There  are  many 
methods  for  data  fusion,  three  methodologies  discussed  in  this  project  including  conventional  method,  Dempster- 
Shafer  method,  and  maximal  method.  The  proposed  methods  mainly  focus  on  how  to  transform  intelligent 
information  into  a  meaningful  picture  of  battlefield  situation.  An  air  campaign  may  have  several  operations,  some 
operations  are  parallel  and  some  operations  are  sequential.  Given  the  probabilities  of  delay  or  success  of  all  or  some 
operations,  the  proposed  methods  attempt  to  draw  the  conclusion  about  the  probability  of  success  of  the  campaign. 
In  addition,  the  proposed  methods  determine  the  total  time  of  delay  if  there  is  delay.  The  conventional  method 
simply  draws  a  conclusion  based  on  largest  probability  of  all  parallel  operations.  The  Dempster-Shafter  theory  fuses 
multiple  probabilities  to  give  conclusion  about  the  probability  of  delay  of  the  campaign.  The  maximal  method  is  a 
classical  statistical  method  in  which  parallel  operations  are  combined  into  an  operation  so  that  all  tasks  are 
sequential  in  order  to  applying  normal  distribution  property. 


MODEL-BASED  ASSESSMENT  OF  CAMPAIGN  PLAN 
PERFORMANCE  UNDER  UNCERTAINTY 

Nong  Ye,  Vuong  Nguyen,  George  Runger 


1-  Introduction 

The  command  and  control  (C2)  of  an  air  campaign  involve  different  processes  (planning,  scheduling,  execution, 
monitoring,  assessment,  and  re-planning)  in  different  functional  areas  (force  application,  force  enhancement,  and 
force  support)  at  different  levels  (from  strategic  to  tactical)  [1-3].  Currently,  C2  operations  suffer  from  a  number  of 
shortfalls  [3].  First,  the  planning  of  an  air  campaign  is  cyclic  by  a  daily  or  weekly  timeline,  which  results  in  a  slow 
and  inefficient  response  to  changes  in  campaign  objectives  and  battlefield  situations  during  plan  execution.  Second, 
there  is  a  lack  of  integration  between  C2  processes.  Third,  the  C2  processes  are  human  intensive.  It  requires  days  for 
planning,  scheduling,  monitoring,  assessment,  and  re-planning.  Therefore,  the  integration  and  automation  of  C2 
operations  is  highly  desirable  to  move  C2  operations  from  schedule-driven  to  situation-driven  continuous  operation. 
Integration  and  automation  improve  the  responsiveness  to  dynamically  changing  campaign  objectives  and  battlefield 
situations  from  days  to  hours  or  minutes. 

Through  the  Summer  Faculty  Research  Program,  an  integration  infrastructure  was  established  to  support 
continuous  C2  operations  based  on  a  process-engineering  schema  for  the  control  and  analysis  of  dynamic  process 
systems  in  industrial  environment  [4].  To  demonstrate  the  feasibility  of  this  processing  engineering  approach  to  the 
integration  of  C2  operations,  a  process  model  was  defined  using  an  air  campaign  scenario.  Research  work  on  the 
automation  of  C2  operations  is  proposed  here  to  provide  a  more  comprehensive  vision  of  the  process  engineering 
approach  to  continuos  C2  operations. 

To  support  integration  and  automation  of  C2  operations,  appropriate  methods  need  to  be  identified  for  all 
processes.  However,  due  to  time  limitation,  only  data  fusion  methods  for  the  monitoring  and  assessment  processes 
are  addressed  in  this  proposal.  Monitoring  and  assessment  processes  can  be  considered  as  an  integrated  process,  one 
updates  the  situation  of  the  air  campaign,  detect  changes  of  campaign  plans  using  Intelligence,  Surveillance,  and 
Reconnaissance  (ISR)  information.  The  other  assesses  the  changes  by  associating  the  current  situation  with  current 
objectives  of  the  campaign  plan.  Together,  they  provide  feedback  to  operation  commander  for  making  decisions 
regarding  the  current  situation  of  the  campaign. 

As  stated  above,  an  air  campaign  plan  can  be  represented  as  a  dynamic  system  in  which  the  performance  of  one 
task  affects  the  performance  of  subsequent  tasks.  A  change  to  a  task  may  associate  with  the  preconditions  or  the 
execution  of  the  task,  which  in  turn  causes  a  change  in  the  start  time,  end  time,  and  duration  of  the  task.  More 
severely,  it  may  change  the  whole  situation  of  the  campaign  plan.  We  need  to  assess  the  performance  of  an  air 
campaign  based  on  ISR  information  regularly  to  determine  whether  the  campaign  plan  should  be  continued,  re¬ 
planned,  or  terminated.  In  other  words,  periodic  campaign  assessment  improves  the  responsiveness  to  dynamic 
battlefield  situations. 

In  our  work,  there  are  three  methods  will  be  utilized  to  fuse  ISR  information  and  assess  the  campaign  plan 
including  conventional  method,  Dempster-Shafer  method,  and  maximal  method.  Section  2  introduces  an  air 
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campaign  scenario  and  the  process  model  of  the  air  campaign.  Section  3  defines  the  meaning  of  confident  interval 
and  measure  of  merits  (MoM),  which  will  be  used  extensively  to  evaluate  the  performance  of  proposed  techniques. 
Section  4  proposes  methods  to  fuse  ISR  information  into  a  meaningful  picture  of  battlefield  situation.  Section  5  tests 
and  analyzes  the  performance  of  those  techniques. 

2.  An  Process  Model  of  an  Air  Campaign 

The  process-engineering  schema  has  traditionally  been  used  to  model  industrial  system  (e.g.  discrete  part  or 
continuous  process  manufacturing  systems)  for  the  planing,  control,  and  analysis  purposes.  An  industrial  process 
system  is  modeled  at  four  levels  of  abstraction:  objective,  conceptual,  functional,  and  physical.  At  the  objective 
level,  the  goals  of  the  systems  are  stated.  At  the  conceptual  level,  the  goals  of  the  systems  are  transformed  into 
operational  definitions  of  system  behavior  in  terms  of  time  phased  and  sequenced  tasks.  At  the  functional  level, 
tasks  at  the  conceptual  level  are  transformed  into  tasks,  which  are  performed  by  functional  units  of  the  system.  At 
the  physical  level,  tasks  at  the  functional  level  are  implemented  by  events  that  are  executed  by  physical  units  of  the 
system.  From  the  objective  level  to  the  physical  level,  the  focus  moves  from  the  global,  abstract  view  of  the  system 
to  the  local,  specific  view  of  system  components  and  their  relationships. 

Similarly,  an  air  campaign  can  be  modeled  at  four  levels,  which  are  objective  level,  conceptual  level, 
functional  level,  and  physical  level.  At  the  objective  level,  campaign  objectives  are  defined  including  start  states, 
intermediate  states,  and  desired  end  states  of  the  air  campaign.  In  addition,  the  objective  level  specifies  constraints  if 
there  are  any.  In  the  campaign  scenario  under  consideration,  the  objectives  of  the  ‘blue1  force  are  to  rescue  a 
delegation  in  a  US  Embassy  under  the  control  of  a  ‘red’  force,  and  to  secure  an  airport  under  the  control  of  the  ‘red’ 
force  (Figure  1).  These  two  objectives  must  be  simultaneously  accomplished  [4]. 


Fig.  L  The  ‘blue’  force  needs  to  rescue  a  delegation  in  a  US  Embassy  and  secures  the  airport  under  the 
control  of  the  ‘red’  force. 

At  the  conceptual  level,  campaign  objectives  are  expended  into  phased  and  sequenced  states.  Conceptual  states 
reflect  situations  of  the  air  campaign  system  at  critical  moments  of  the  campaign  plan.  Integrated  joint  force  tasks 
are  not  segregated  into  different  functional  areas  such  as  force  application  (FA),  force  enhancement  (FE),  and  force 
support.  However,  the  blue  force  is  refined  into  smaller  groups,  which  can  be  US  force  and  allied  force,  and  their 
tasks. 

The  strategic  plan  of  the  air  campaign  at  the  conceptual  level  is  supported  by  a  tactical  plan  of  the  functional 
level.  Tasks  of  the  US  force  and  the  allied  force  are  further  decomposed  into  tasks  of  functional  areas  (FA,  FE,  and 
FS). 
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At  the  physical  level,  tactical  plan  at  the  functional  level  is  further  refined  into  a  tactical  plan  dealing  with 
execution  of  each  functional  component.  Components  and  tasks  are  physically  identified.  They  are  mapped  in  their 
raw  forms  such  as  images  of  aircraft,  roads,  bridges,  etc. 

In  our  work,  we  focus  on  an  air  campaign  plan  at  the  physical  level.  The  air  campaign  process  is  represented 
by  a  sequence  of  tasks  in  which  the  delay  of  a  task  will  result  in  the  delay  of  the  starting  time  of  subsequent  tasks. 
However,  the  delay  of  a  task  does  not  affect  the  duration  of  other  tasks  dues  to  our  task  independent  assumption. 
Figure  2  shows  the  tasks  of  US  force  and  the  expected  duration  (jlx)  of  tasks.  In  addition,  the  standard  deviation  (a) 
of  a  task  is  specified  to  be  1/8  of  its  duration. 


Task  G: 

|  Establish 
Security 
zone 

|  (15  min) 


Task  H: 
Secure 
air  force 
(10  min) 


Fig.  2.  Activities  of  US  force  at  the  physical  level.  Numbers  on  the  top  of  boxes  indicate  the  decision 
time  when  ISR  information  arrives.  Number  inside  the  boxes  indicates  the  expected  duration  of  tasks. 


3.  The  Definition  of  MoM  and  Confidence  Value 

A  measure  of  merits  (MoM)  of  a  task  is  a  composite  index  that  indicates  the  degree  of  satisfaction  of  achieving  that 
task.  A  negative  MoM  indicates  that  the  task  is  accomplished  earlier  than  the  expectation.  A  positive  MoM  indicates 
a  delay  in  accomplishing  the  task.  A  MoM  value  of  zero  indicates  that  the  task  is  accomplished  right  on  time. 
However,  providing  the  MoM  index  without  a  confidence  level  is  meaningless.  A  confidence  level  indicates  the 
degree  of  belief  in  a  delay  of  accomplishing  task.  For  instance,  at  the  decision  time  1,  ISR  information  report  that 
task  A  may  be  delayed  for  4  minutes  with  80  percent  confidence  (MoM  =  4,  confidence  =  80%).  Task  1  may  be 
delayed  for  -1  minute  with  90  percent  confidence  (MoM  =  -1,  confidence  -  90%).  Based  on  this  information, 
proposed  methods  make  inference  and  recommendation  about  the  whole  campaign  plan  to  determine  whether  the 
campaign  needs  to  be  modified.  As  the  time  goes  on,  ISR  information  of  more  tasks  are  available,  the  accuracy  of 
campaign  assessment  will  increase.  If  the  ISR  information  reports  that  task  A  cannot  be  completed  at  all  due  to  some 
reasons  (MoM  =  oc  at  100%  confidence  level),  then  the  air  campaign  plan  should  be  terminated  or  re-planned. 


4.  Information  Fusion  Techniques 

There  are  various  types  of  technique  dealing  with  reasoning  under  uncertainty,  including  statistical  methods  (e.g., 
hypothesis  testing),  probabilistic  methods  (e.g.,  Bayes’  rule,  Bayesian  networks),  certainty  factor,  Dempster-Shafer 
theory,  and  fuzzy  logic  [5-7].  Many  studies  have  proposed  a  stand-alone  technique  to  deal  with  uncertainty 
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situation.  We  develop  hybrid  methods  those  utilize  both  classical  technique  and  Dempster-Shafer  for  information 
fusion.  The  proposed  methods  address  the  problem  in  two  steps.  The  first  step  will  be  the  interpretation  of  ISR 
information.  There  are  three  possible  interpretations  for  raw  ISR  information,  including  discrete  distribution, 
conservative  continuous  distribution  and  optimistic  continuous  distribution.  The  second  step  is  to  apply  the 
proposed  methods  to  the  processing  ISR  information  to  assess  the  battlefield  situation. 

Consider  the  following  example.  ISR  reports  that  task  A  (MoM  =  1,  confidence  =  95%),  task  B  (MoM  =  1, 
confidence  =  98%),  task  C  (MoM  =  -5,  confidence  =  90%),  and  other  tasks  are  unknown.  Since  there  is  no 
information  for  other  tasks,  their  duration  assume  remaining  the  same.  We  will  use  this  example  to  demonstrate  the 
proposed  methods  for  all  three  interpretations. 


4.1.  Discrete  Distribution 

If  the  duration  of  tasks  follow  discrete  distribution,  the  new  duration  of  tasks  A,  B,  and  C  are  updated  for  the  given 
example  as  follows. 


JlO  (5%)  (15  (2%) 

[It  (95%)  [16  (98%) 


JlO  (10%) 
[5  (90%) 


The  duration  of  other  tasks  remains  the  same  as  given  in  Figure  2.  Figure  3  shows  the  discrete  distribution  of 
task  A.  Tasks  B  and  C  can  be  represented  in  the  same  manner. 


% 

95 

5 

10  11  minute 

Fig.  3.  Discrete  distribution:  Task  A  will  delay  1  minute  at  95  percent  confidence,  and  will  be  on  time  at 
5  percent  confidence  level  [(11,  0.95),  (10,  0.05)]. 

The  process  model  given  in  Figure  2  can  be  decomposed  into  two  paths:  path  1  includes  tasks  A,  B,  D,  E,  F,  G, 
and  H,  path  2  includes  tasks  A,  C,  D,  E,  F,  G,  and  H.  In  current  example,  we  know  two  tasks  for  both  path  1  (tasks  A 
and  task  B)  and  path  2  (task  A  and  task  C),  therefore,  we  have  four  combinations  for  each  path.  Among  4 
combinations,  whichever  has  total  duration  greater  than  90  (the  expected  duration  of  the  campaign  plan),  their 
probability  will  be  summed  up  to  calculate  the  probability  of  delay.  The  total  of  operation  duration  and  probability 
of  delay  for  the  two  paths  are  calculated  by  following  equation 

pt=Yad^  where  p=Y\pj  w 

*  j 

Note  that  p,  is  the  duration  of  tasks.  PT  is  the  probability  of  delay  of  the  operation,  and  P  is  the  probability  of 
delay  of  given  combination. 

From  equation  (1),  the  total  operation  time  and  confidence  level  of  path  1  are 

Combination  1:  task  A  is  NOT  delayed  and  task  1  is  NOT  delayed. 
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pT  =  10  +  15  +  10  +  15  +  15  +  15  +  10  -  90  (<90) 

P  =  0.05  *  0.02  =  0.001 

Combination  2:  task  A  is  NOT  delayed  but  task  1  is  delayed 

pT  -  10  + 16  +  10  +  15  +  15  +  15  +  10  =  91  (>90) 

P  =  0.05  *  0.98  =  0.049 

Combination  3:  task  A  is  delayed  but  task  1  is  NOT  delayed 

pT  =  U  + 75  +  10  +  15  +  15  +  15  +  10  =  91  (>90) 

P  =  0.95  *  0.02  =  0.019 

Combination  4:  task  A  is  delayed  and  task  1  is  delayed. 

|iiT  =  77  +  76+10+15  +  15  +  15  +  10  =  92  (>90) 

P  =  0.95  *  0.98  =  0.931 

Combinations  2,  3,  and  4  have  pT  greater  than  90;  therefore,  are  summed  up  to  calculate  the  MoM  and  the 
probability  of  delay  of  path  1  are: 

MoM  =  (91-90)*0.049  +  (91-90)*0.019+(92-90)*0.93 1=1 .93 
P  =  IP  (Ds)  =  0.049  +  0.019  +  0.931  =  0.999 

For  path  1,  its  MoM  and  confidence  level  are  calculated  in  similar  manner,  which  yields  following  results 
MoM  =  (85-90)*0.05+(80-90)*0.045+(86-90)*0.095+(81-90)*0.855=-8.775 

P  =  IP  (Di)  =  0% 

Having  the  probability  of  delay  of  two  paths,  we  can  calculate  the  probability  of  delay  for  the  whole  campaign 
plan  by  using  the  proposed  methods. 

Conventional  Method:  Conventionally  we  just  select  the  maximum  probability  of  delay  of  two  paths.  In  this 
example,  we  conclude  that  the  campaign  will  be  delayed  for  1.93  minutes  at  99.9%  confidence  level. 
Dempster-Shafer  Method:  Equation  2  is  a  Dempster’s  rule  of  combination,  which  can  be  used  to  combine  the 
probability  of  delay  of  the  operations. 

Y,mx{X)m2{Y)  (2) 

m.  0  m2  (Z)  =  - 

1  2  1  -K 


Table  1.  Combining  probability  of  delay  of  US  operation  and  allied  operation 


m2({D})=0 

m2(0)=l 

m1({D})=0.999 

ml0m2({D})=O 

m1®m2({D})=0.999 

mi(©)=0.001 

mi®m2({D})=0 

ml©m2(©)=0.001 

Where  mj({D})  is  the  mass  of  evidence  of  delay  of  path  1 

m^©)  is  the  degree  of  disbelief  in  evidence  of  delay  of  path  1 

m2({D})  is  the  mass  of  evidence  of  delay  of  path  2 

m2(0)  is  the  degree  of  disbelief  in  evidence  of  delay  of  path  2 
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Using  Dempster_Shafer's  rule  of  combination,  the  probability  of  delay  of  the  campaign  is 

m]©m2({D})  =  0.999  +  0  +  0 
-0.999 

We  can  conclude  that  the  campaign  will  be  delayed  for  1.93  minutes  at  99.9  percent  confidence  [7,  8,  9,  10] 

4.2.  Continuous  Distribution:  Optimistic  Interpretation. 

In  optimistic  interpretation,  task  A  will  be  accomplished  between  10  minutes  and  11  minutes  at  95  percent 
confidence  level  [P  ( 10  <  A  <  1 1)  =  0.95]  (Figure  4). 


Fig.  4.  Optimistic  interpretation  for  task  A 


New  mean  (p)  and  standard  deviation  (a)  of  tasks  need  to  be  re-calculated.  There  are  several  ways  to  calculate 
new  p.  In  our  work,  new  p  will  be  calculated  as  the  average  of  two  given  values.  For  task  A,  pA  is  10.5,  which  is  the 
middle  of  10  and  1 1 .  Since  we  know  pA  and  probability  (P-0.95),  we  can  calculate  aA. 


x-m 


X-v 


11-10.5 


:  0.2551 


a  Z  Z(P  =  0.975) 

In  the  same  manner,  the  new  p  of  tasks  B  and  C  are  15.5,  and  7.5,  respectively.  The  new  a  of  tasks  A  is  0.2149 
and  task  B  is  1 .5199.  Since  there  is  no  new  information  of  other  tasks,  their  p  and  a  remain  the  same. 

The  total  of  operation  duration  and  its  standard  deviation  of  distribution  for  path  1  and  path  2  are  calculated 
based  on  the  probability  of  normal  distribution  [11] 


Mr  =2>, 


r=  2>-2 


(3) 


From  Equation  3,  new  pT  and  aT  of  path  1  are 

fiT  =  juA  +  fiB  +  jjp  +  fig  +  fig  +  fig  -f  fiH  =  1 0.5  +  15.5  +  10  +  15  +  15  +  15  +  10  =  91 
<7r  =  V0.255  +  0.215  +  1.563  +  3.516  +  3.5 16  +  3.5 16  + 1.563  =  3.7126 
Similarly,  new  pT  and  gt  of  path  2  are 

fij  —  fiA  +  fic  +  fiD  +  fig  +  fig  +  fiQ  +  fig  —  1 0.5  +  7.5  +  10  +  15  +  15  +  15  +  10  ~  83 
crT  =  V0.255  + 1.520+ 1.563  +  3.516  +  3.5 16  +  3.516  + 1.563  =  4.006 
Given  distribution  of  the  operations,  we  can  calculate  the  probability  that  the  campaign  will  be  delayed.  From 
Figure  5,  the  probability  of  delay  for  path  1,  path  2  is  0.606,  0.04,  respectively. 
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90  91 


83  90 


Fig.  5.  Distribution  of  path  1  and  path  2,  respectively 

Given  the  probabilities  of  delay  of  the  two  paths,  we  can  calculate  the  probability  of  delay  for  the  whole 
campaign  plan  by  using  the  proposed  methods. 

Conventional  Method:  Perform  calculation  as  in  discrete  distribution,  we  conclude  that  the  campaign  plan  will  be 
delayed  for  1  minute  at  60.6  percent  confidence. 

Dempster-Shafer  Method:  Perform  calculation  as  in  discrete  distribution,  we  conclude  that  the  whole  campaign 
plan  will  be  delayed  for  1  minute  at  62.2  percent  confidence. 


Maximal  Method:  Instead  of  decomposing  the  campaign  (Figure  2)  into  two  paths,  the  maximal  method  combines 
parallel  tasks  into  single  task  and  applies  the  property  of  the  normal  distribution  to  inference  the  distribution  of  the 
whole  campaign.  In  this  scenario,  task  B  and  C  are  parallel,  the  maximal  method  select  the  maximum  between  these 
two  tasks,  called  task  Y  (Figure  6)  by  using  equation  4. 


Task  Y: 

xxxxx 

■1 

Task  D: 
Prepare 
for 

landing 
i  (15  min) 

m 

Task  E: 
Land 

more 

force 
(15  min) 

■ 

Task  F: 
Load  off 
force 
(15  min) 

■ 

Task  G: 
Establish 
Security 
zone 
(15  min) 

■ 

(x  min) 

■ 

m 

■ 

s 

■ 

Task  H: 
Secure 
air  force 
(10  min) 


Fig.  6.  Using  maximal  method  to  combine  two  parallel  tasks  into  single  task 


F(y)  =  P(z  =  max(£,  C)<  y)  =  P(B  <  y)  *  P(C  <  y) 
dP  dP  dP 

Ay)  =^  =  P(B<y)^-  +  ^P(C<y) 

oy  oy  oy 


(4) 


Ay)  =  ® 


y_z±i 

\  aB  J 


fv(y)  +  ® 


y-  Me 


\ 


V  J 


'fc(y) 


Then  task  Y  will  be  combined  with  task  X,  which  is  the  sum  of  serial  tasks  (X  —  A  +  C  +  D+  E  +  F  +  G  +  H) 
using  Equation  5. 
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(5) 


P(X  +  Y  <  90)  =  EP(X  +  Y  <9Q\Y  =  y) 

P(X  +  Y  <  90)  =  EyP(X  <  C  -  y) 

P{X  +  Y  <  90)  =  £‘<bf  90  -  -  -  Mx  *  f(y)dy 
38  {.  ax  J 

It  is  reasonable  to  assume  the  range  of  any  task  will  be  its  mean  plus  3  standard  deviations.  Therefore,  the 
maximum  and  minimum  value  of  task  B  will  most  likely  be  /JB  ±  3<JB ,  which  are  20.6  for  maximum  value  and 
9.38  for  minimum  value.  These  two  values  will  be  used  as  upper  and  lower  bound  for  the  integration  in  the  equation 
Equation  5  yields  the  probability  of  not  delay,  so  the  probability  of  delay  will  be  1  -  P(X  +  Y  <  90).  Using 
equation  4  and  5,  the  probability  of  delay  of  this  example  is  60.62  %  and  MoM  is  1.93  minutes. 

4.3.  Continuous  Distribution:  Conservative  Interpretation. 

In  a  conservative  interpretation,  task  A  needs  at  least  1 1  minutes  to  be  done  at  95  percent  confidence  [P  (A  >  11)- 
0.95]  (Figure  7).  Other  tasks  can  be  interpreted  in  the  same  manner. 


Considering  task  A  again,  since  we  know  its  probability  and  its  a,  we  can  calculate  ji 
Z  =  =>^  =  A'-cr*Z(/5  =  l-  0.95)  =  1 1  -  -  •  1 1  *  (- 1 .6449)  =  13.262 

G  8 

In  similar  manner,  new  p  of  tasks  1  and  2  are  20.107  and  5.801,  respectively.  The  a  of  tasks  A  is  1.38,  task  1  is 
2,  and  task  2  is  0.63.  p  and  a  of  other  tasks  remain  the  same. 

How  to  obtain  the  pT  and  aT  for  the  distributions  of  path  1  and  path  2  is  similar  to  the  optimistic  interpretation 
case.  From  Equation  3,  the  pT  and  aT  of  path  1  are  98.4  and  4.423,  respectively.  The  pT  and  aT  of  path  2  are  84  and 
3.994,  respectively. 

Similar  to  optimistic  interpretation,  given  distribution  of  both  paths,  we  can  calculate  the  probability  that  the 
campaign  will  be  delayed.  The  probabilities  of  delay  of  path  1  and  path  2  are  0.9708,  0.0686,  respectively  (Figure 
8).  From  these  probabilities,  the  probability  of  the  whole  campaign  plan  can  be  obtained  using  proposed  methods. 

gt  =4.423 

90  98.4  84  90 

Fig.  8.  Distribution  of  path  1  and  path  2,  respectively 
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Conventional  Method:  Similar  to  calculation  in  discrete  distribution,  we  conclude  that  the  whole  campaign  plan 
will  be  delayed  for  8.4  minutes  at  97.08  percent  confidence. 

Dempster-Shafer  Method:  Similar  to  calculation  in  discrete  distribution,  we  conclude  that  the  whole  campaign 
plan  will  be  delayed  for  8.4  minutes  at  97.28  percent  confidence 

Maximal  Method:  Similar  to  calculation  in  optimist  interpretation,  we  conclude  that  the  whole  campaign  plan  will 
be  delayed  for  1.93  minutes  (or  2  minutes)  at  83.28  percent  confidence 

5.  Testing  and  Results 

We  create  a  numbers  of  testing  cases  for  the  air  campaign  scenario  in  Figure  2  to  exam  the  performance  of  the 
proposed  techniques.  Cases  can  be  classified  into  different  types  based  on  the  number  of  task  that  ISR  reports  ahead 
of  time  at  each  decision  time.  ISR  can  report  the  MoM  and  probability  of  delay  of  one  task,  two  tasks,  or  all  tasks 
ahead  of  time.  In  this  project,  three  types  of  testing  cases  are  generated.  Type  1  is  test  case  for  reporting  one  task 
ahead  of  time.  Similarly,  type  2  and  type  3  is  test  case  for  reporting  two  tasks  and  three  tasks  ahead  of  time, 
respectively.  There  are  five  scenarios  for  each  type  of  test  case.  Appendixes  1,  2,  and  3  are  scenarios  for  reporting  1, 
2,  and  3  tasks  ahead  of  time,  respectively. 

Table  2  shows  several  scenarios  of  type  1  test  case.  Consider  scenario  1  at  time  1  (Tl),  ISR  only  reports  the 
MoM  and  the  probability  of  delay  for  task  A,  which  is  MoM  =  5  and  confidence  =  77  %.  Information  of  other  tasks 
is  unknown.  At  T2,  probability  of  task  A  is  100%  because  task  A  have  already  completed.  From  this,  task  A  is 
accomplished  5  minutes  earlier  than  expectation.  Task  B  and  C  predicted  to  be  completed  earlier  than  expected  by  4 
minutes  at  79  %  confidence  and  5  minute  at  88  %  confidence,  respectively.  As  time  is  moving  on,  more  tasks  will  be 
known  with  certainty,  and  so  the  campaign.  For  instance,  at  T7  the  probability  of  delay  for  the  whole  scenario  can  be 
predicted  without  much  variation  since  tasks  A,  B,  C,  D,  E,  F,  and  G  are  known  with  certainty,  but  task  H. 

5.1.  Results 

For  each  decision  time  of  a  testing  case,  we  use  the  proposed  methods  to  determine  the  MoM  and  the  confidence 
value  for  the  whole  campaign  scenario  at  that  decision  time.  This  pair  of  MoM  and  confidence  values  is  then 
compared  with  the  pair  of  MoM  and  confidence  values  obtained  from  the  simulation  that  we  assume  represents  the 
real  situation  of  the  plan  execution.  Three  interpretations  represent  totally  different  populations;  therefore,  three 
simulations  will  be  created  for  three  corresponding  proposed  interpretations.  Appendixes  4,  5,  and  6  are  results  of 
type  1,  type  2,  and  type  3  test  cases  for  discrete  distribution.  Appendixes  7,  8,  and  9  are  results  of  type  1,  type  2,  and 
type  3  test  case  for  optimist  interpretation.  Appendixes  10,  11,  and  12  are  results  of  type  1,  type  2,  and  type  3  test 
case  for  conservative  interpretation. 
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Table  2.  Several  test  cases  for  type  1:  reporting  one  task  ahead  of  time 
SCENARIO  1  SCENARIO  2  SCENARIO  3  SCENARIO  4  SCENARIO  5 


MoM 

% 

MoM  % 

MoM 

% 

MoM 

% 

MoM 

% 

A 

4 

77 

-2  93 

1 

71 

2 

79 

1 

78 

B 

c 

Time  1 

D 

E 

F 

G 

H 

MoM 

% 

MoM  % 

MoM 

% 

MoM 

% 

MoM 

% 

A 

-5 

100 

0  100 

1 

100 

-4 

100 

-5 

100 

B 

-4 

79 

5  71 

1 

99 

-2 

99 

-2 

87 

c 

-5 

88 

-2  99 

-4 

75 

-3 

73 

5 

86 

Time  2 

D 

E 

F 

G 

. 

H 

. 

. 

. 

* 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

MoM 

% 

MoM  % 

MoM 

% 

MoM 

% 

MoM 

% 

A 

-5 

100 

0  100 

1 

100 

-4 

100 

-5 

100 

B 

3 

100 

2  100 

-1 

100 

-3 

100 

-5 

100 

C 

1 

100 

-4  100 

-3 

100 

-5 

100 

-2 

100 

Time  7 

D 

5 

100 

1  100 

-4 

100 

-2 

100 

-4 

100 

E 

-2 

100 

-5  100 

-1 

100 

-4 

100 

-4 

100 

F 

-1 

100 

5  100 

5 

100 

-1 

100 

3 

100 

G 

-1 

100 

-1  100 

3 

100 

5 

100 

0 

100 

H 

1 

77 

3  74 

-1 

94 

4 

84 

-1 

85 

5.2.  Performance  Analysis 

From  the  plots,  the  proposed  methods  have  demonstrated  theirs  superior  performance  in  determining  the  probability 
of  delay  of  the  campaign  plan.  Methods  for  determining  of  MoM  need  to  be  refined  for  more  accuracy  (see 
Appendixes  4-12).  Among  three  interpretations,  discrete  interpretation  has  shown  better  results  than  the  others. 
However,  the  discrete  method  requires  more  calculation  than  the  others.  Among  proposed  methods,  the  Dempster- 
Shafer  and  the  conventional  method  have  relatively  shown  theirs  superior  to  the  maximal  method  although  the 
calculation  of  maximal  method  is  more  complex  than  of  the  conventional  method  and  the  Dempster-Shafer  theory 
method.  Therefore,  the  Dempster-Shafer  and  the  conventional  method  are  recommended  for  future  use. 
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Graphical  Comparison  of  Confidence  Values  of  Proposed  Methods  and  Simulation  for  Discrete  Distribution 
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Graphical  Comparison  of  Confidence  Values  of  Proposed  Methods  and  Simulation  for  Conservative  Interpretation 
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Continuous-Conservative-Type  3-Scenario  5 
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6.  Conclusion 

The  paper  presents  the  methods  that  assess  the  performance  of  an  air  campaign  plan  based  on  ISR  information  to 
determine  whether  the  campaign  should  be  continued,  re-planned,  or  terminated.  The  methods  help  improving  the 
responsiveness  to  dynamic  battlefield  situations  by  giving  recommendations  to  the  commander  of  the  operation. 
Based  on  these  recommendations,  the  commander  can  determine  what  need  to  be  done  to  accomplish  the  air 
campaign. 
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Appendix  1. 


Scenarios  for  predicting  one  task  ahead  of  time 


SCENARIO  1  SCENARIO  2  SCENARIO  3  SCENARIO  4  SCENARIO  5 

MoM  %  MoM  %  MoM  %  MoM  %  MoM  % 
A  4  77  -2  93  1  71  2  79  1  78 

B 
C 

Time  1  D 
E 
F 
G 
H 


MoM 

% 

MoM 

% 

MoM 

% 

MoM 

% 

MoM 

% 

A 

-5 

100 

0 

100 

1 

100 

-4 

100 

-5 

100 

B 

-4 

79 

5 

71 

1 

99 

-2 

99 

-2 

87 

C 

-5 

88 

-2 

99 

-4 

75 

-3 

73 

5 

86 

Time  2  D 
E 
F 
G 
H 


MoM  %  MoM  %  MoM  %  MoM  %  MoM  % 


A  -5  100 

B  3  93 

C  1  94 

Time  3D  3  76 

E 
F 
G 
H 


0  100  1  100 

2  100  -1  70 

-4  73  -3  97 

-2  76  -4  80 


■4 

100 

-5 

100 

•3 

78 

-5 

91 

■5 

85 

-2 

71 

3 

95 

-1 

96 

MoM  %  MoM  %  MoM  %  MoM  %  MoM  % 


A  -5  100 

B  3  100 

C  1  100 

Time  4  D  5  93 

E  1  83 

F 
G 
H 


0 

100 

1 

100 

2 

100 

-1 

100 

-4 

100 

-3 

100 

1 

74 

-4 

94 

4 

89 

-5 

81 

-4 

100 

-5 

100 

-3 

100 

-5 

100 

-5 

100 

-2 

100 

-2 

85 

-4 

73 

0 

99 

-2 

71 

MoM 

% 

MoM 

% 

MoM 

% 

MoM 

% 

MoM 

% 

A 

-5 

100 

0 

100 

1 

100 

-4 

100 

-5 

100 

B 

3 

100 

2 

100 

-1 

100 

-3 

100 

-5 

100 

C 

1 

100 

-4 

100 

-3 

100 

-5 

100 

-2 

100 
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Time  5 

D 

5 

100 

1 

100 

-4 

100 

-2 

100 

-4 

100 

E 

-2 

74 

-5 

71 

-1 

98 

-4 

84 

-4 

86 

F 

5 

89 

2 

75 

4 

71 

-2 

89 

-4 

88 

G 

H 

MoM 

% 
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% 
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% 
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% 
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-5 
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3 
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-1 
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-3 
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-5 
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-4 
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-2 
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-4 
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-4 
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F 

-1 

89 

5 

79 

5 

100 

-1 

95 

3 

74 

G 

-2 

84 

-5 

81 

2 

96 

5 

97 

5 

70 

H 

MoM 

% 
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% 

MoM 

% 

MoM 

% 

MoM 

% 

A 

-5 

100 

0 

100 

1 

100 

-4 

100 

-5 

100 
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3 

100 

2 

100 

-1 

100 

-3 

100 

-5 

100 

C 
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100 

-4 

100 

-3 

100 

-5 

100 

-2 

100 

Time  7 
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5 

100 

1 

100 

-4 

100 

-2 

100 

-4 

100 
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-2 

100 

-5 

100 

-1 

100 
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100 
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100 
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100 
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100 

-1 

100 

3 

100 
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-1 

100 

-1 

100 

3 

100 

5 

100 

0 

100 
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77 

3 

74 

-1 

94 

4 

84 

-1 

85 

1 1-20 


Appendix  2. 


Scenarios  for  predicting  two  tasks  ahead  of  time 


Time  1 


SCENARIO  1  SCENARIO  2  SCENARIO  3  SCENARIO  4  SCENARIO  5 


MoM 

% 

MoM 

% 

MoM 

% 

MoM 

% 

MoM 

% 

A 

1 

95 

5 

88 

-5 

76 

4 

95 

-5 

73 

B 

1 

98 

-2 

70 

-1 

90 

5 

97 

4 

75 

C 

-5 

90 

1 

87 

1 

94 

-2 

73 

3 

81 

Time  2 


MoM 

% 

MoM 

% 

MoM 

% 

MoM 

% 

MoM 

% 
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0 

100 

4 

100 

-1 

100 

3 

100 

-5 

100 

B 

-4 

97 

-5 

90 

3 

77 

-2 

100 

-3 

76 

C 

-4 

93 

2 

95 

-3 

75 

5 

87 

5 

95 

D 

-5 

80 

5 

71 

1 

81 

2 

79 

-2 

78 

Time  3 
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% 

MoM 

% 

MoM 

% 

MoM 

% 

MoM 

% 

A 

0 

100 

4 

100 

-1 

100 

3 

100 

-5 

100 

B 

-2 

99 

3 

83 

-3 

93 

-2 

100 

2 

83 

C 

2 

94 

5 

100 

2 

76 

-3 

81 

1 

93 

D 

1 

100 

5 

84 

-1 

99 

3 

74 

-2 

90 

E 

1 

83 

2 

86 

-2 

81 

-2 

78 

2 

80 
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% 

MoM 

% 

MoM 

% 

MoM 

% 

MoM 

% 
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0 

100 

4 
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-1 

100 

3 

100 

-5 

100 

B 

-2 

100 

3 

100 

-3 

100 

-2 

100 

2 

100 

C 

2 
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5 

100 

2 
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-3 

100 

1 

100 

D 

1 

100 

-5 

82 

-2 

93 

3 

85 

5 

80 

E 

-3 

71 

5 

98 

5 

100 

-2 

78 

-5 

86 

F 

-3 

71 

1 

75 

-2 

96 

1 

78 

-1 

90 

G 
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% 
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% 
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% 
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% 
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A 

0 
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-1 
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3 
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-5 
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B 

-2 
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3 
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-3 
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-2 
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2 
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C 

2 
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5 
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-3 

100 

1 
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Appendix  3. 


Scenarios  for  predicting  three  tasks  ahead  of  time 
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Appendix  4.  Results  of  Discrete  Interpretation:  Type  1 
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Appendix  5.  Results  of  Discrete  Interpretation:  Type  2 
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Appendix  6.  Results  of  Discrete  Interpretation:  Type  3 
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Appendix  7.  Result  of  Optimist  Interpretation:  Type  1 
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Appendix  8.  Result  of  Optimist  Interpretation:  Type  2 
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Appendix  9.  Result  of  Optimist  Interpretation:  Type  3 
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Appendix  10.  Results  of  Conservative  Interpretation:  Type  1 
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Appendix  1 1 .  Results  of  Conservative  Interpretation:  Type  2 
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Appendix  12.  Results  of  Conservative  Interpretation:  Type  3 
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